Parametric path modeling for an optical automatic seam tracker and real time robotic control system

ABSTRACT

A control processor in a robotic system treats optically sensed locations along the path ahead of the tool, which are classified in relation to elapsed distances and identified in 3-D coordinates and 3-axes tool orientation, to apply in real time and under feedforward a control signal representing the error between anticipated tool position and anticipated sensed location on the path. In the process, the anticipated sensed location is obtained by reference to a model made ahead of time with the stored sensed locations grouped as a function of elapsed distances and having a common algebraic feature, such as the slope or the change of slope. A taught path for the robot has been recovered by the control processor also based on elapsed distances and control is related to actual tool position provided by the robot.

This application is a continuation-in-part of application Ser. No.07/140,261, filed Dec. 31, 1987, now abandoned, by the present inventorsand owned by the present assignee.

CROSS-REFERENCED PATENT APPLICATIONS

The present invention is related to:

(1) copending patent application Ser. No. 7-139,890 filed Dec. 31, 1987for "Optical Automatic Seam Tracker and Real Time Control System for anIndustrial Robot" which is directed to the hereinafter referred tocontrol processor invention (53,652);

(2) copending patent application Ser. No. 7-140,858 filed Dec. 31. 1987for "Image Processing System for an Optical Seam Tracker" which isdirected to the hereinafter referred to image processor invention(54,150).

TECHNICAL FIELD

The invention relates to robot control for tracking with an effector enda time-based trajectory, in general, and more particularly to a controlsystem for an industrial robot operative with a look-ahead opticalsensor. The invention is applicable to welding, cutting andsealant/adhesive deposition on a seam or joint.

BACKGROUND OF THE INVENTION

The present invention is directed to parametric modeling for an opticalseam tracker and real time robotic control system involving analternative method than the interpolation method described alsohereinafter, as used to select the sensed location corresponding on thesensed seam path to the anticipated tool position for control.

The invention will be described, for illustration, in the context of arobotic arc-welding seam tracking system. It involves two aspects:first, the automatic guidance of the robot, or other manipulator of aneffector end, to follow the seam path as sensed ahead of the tool, andsecondly, image processing conceived and implemented for the detectionof the seam path and for the generation of parametric information to beused in controlling the industrial robot.

It is known from "Progress In Visual Feedback For Robot Arc-Welding OfThin Sheet Steel" by W. F. Clocksin, P. G. Davey, C. G. Morgan and A. R.Vidler, in Proceedings of the 2nd Int. Conf. on Robot Vision and SensoryControl, 1982, pp. 189-200:

(1) to generate with a laser beam a light stripe on the seam to betracked, and to derive with a camera a stripe image in a two-coordinateplane, then, to use triangulation to represent in a three-coordinatesystem a recognizable representation of the joint to be welded;

(2) to compute and store digitally in a data base the informationassociated with each point identified by a stripe along the seam;

(3) to use optically derived data to make a model defining robotpositioning and to derive between triangulated optically and digitallyderived information errors regarding the actual position of the gap ofthe joint, the root corner of the joint, and any standoff as well aslateral errors; and

(4) to use such errors for correction in controlling a welding torch.

However, in the afore-stated reference, optical imaging and jointrecognition are effected separately from torch control in the course oftwo successive seam paths, the second only being used for effectivewelding.

It is known from "Adaptive Robotic Welding Using A Rapid ImagePre-Processor" by M. Dufour and G. Begin, in Robot Vision and SensoryControls, 3rd Conf. Cambridge, Mass., Nov. 6-10, 1983, pp. 641-648 toeffectuate visual sensing and welding simultaneously, whileaccomplishing adaptive robotic welding. Real time control of the robotis the object. However, the article contemplates feedback control of therobot in response to the information obtained by optical sensing.

It is known, from "Tracking Control System For Arc Welding Using ImageSensor" by M. Kawahara and H. Matsui in Proceedings of the EighthTriennial World Congress of the Intern. Fed. of Automatic Control,Kyoto, Japan, Aug. 24-28, 1981 (Oxford, England; Pergamon Press 1982 -pp. 2117-2122 vol. 4), to use data derived from a seam stripe image forjoint recognition and to compute the groove center position from whichto establish groove pattern recognition and torch positioning control.Optical detection is made ahead of the torch, and the target valueobtained from such optical data is delayed at each particular point, tobe applied for control at a time the torch reaches the optically sensedpoint.

This prior art, however, assumes a simple correlation between the twodisplaced locations, for sensing and welding, respectively, notencountered in practice with an industrial robot, where six degrees offreedom and a relatively complex seam are to be operated on.

It is known, from "A Visual Sensor For Arc-Welding Robots" by TakaoBamba, Hisaichi Maruyama, Eiichi Ohuo and Yashunori Shiga inProceedings, 11th Intern. Symposium on Industrial Robots, Tokyo, Japan,1981, pp. 151-158:

(1) to pass a slit beam of light upon the track to be followed so as toderive a series of stripes;

(2) to derive from a stripe by triangulation two-dimensional image ofthe joint;

(3) to detect an horizontal deviation of the robot position from thecenter in the sensing plane;

(4) to feed to a robot control system the deviation signal forcorrection of the torch path.

In this prior art, control of the robot is not analyzed and no othersolution is proposed than under conventional feedback operation.

It is known from: "A Visual Seam Tracking System for Arc-Welding Robots"by T. Bamba, H. Muruyama, N. Kodaira and E. Tsuda presented at the 14thIntern. Symposium on Industrial Robots, Gothenburg, Sweden, Oct. 2-4,1984, pp. 365-373, to provide visual guidance in controlling anarc-welding robot having five degrees of freedom. Sensing, however, iseffected by scanning the joint area cyclically while rotating thesensing unit about the welding torch, and control of the robot isimplemented directly without consideration of feedforward.

It is known from, "Model Driven Vision To Control A Surface FinishingRobot" by D. Graham, S. A. Jenkins and J. R. Woodwark presented at RobotVision and Sensory Controls, 3rd Conf. Cambridge, Mass., Nov. 6-10,1983, pp. 433-439, to control a robot under six degrees of freedom inresponse to a camera image treated and interpreted by a computer. There,however, the problem of real time control has been solved by the use ofoff-line analysis and the determination of preplanned paths, as well asof computer models, for the determination of the robot kinematics.

It is known from "Data Processing Problems For Gas Metal Arc (GMA)Welder" by G. Nachev, B. Petkov, and L. Blagoev presented at RobotVision and Sensory Controls, 3rd Conf. Cambridge, Mass., Nov. 6-10,1983, pp. 675-680, to detect five characteristic points of the weldjoint to be used in an optical sensor coordinate system and to providetherefrom information using coordinate transformations for control of anindustrial robot. This approach, however, assumes an adaptation of therobot to such implementation, rather than providing control commandsexternally from the robot inherent characteristics.

It is generally known, from "A Real-Time Optical Profile Sensor ForRobot Arc Welding" by G. L. Oomen and W. J. P. A. Verbeek, presented atRobot Vision and Sensory Controls, 3rd Conf. Cambridge, Mass , November,6-10, pp. 659-668, to derive from an optical sensing system coordinatesof the path to be followed by the torch, using transformation fromcamera coordinates to workpiece coordinates to place with a robotcontroller the torch in correct position How this is done has not beshown, and the problems there involved were not considered, norsolutions were given.

It is generally known from "Joint Tracking and Adaptive Robotic WeldingUsing Vision Sensing of the Weld Joint Geometry" by J. E. Agapakis, J.M. Katz, M. Koifman, G. N. Epstein, J. M. Friedman, D. O. Eyring and H.J. Rutishauser in Welding Journal, November 1986, pp. 33-41 to usevision processing with stripe extraction to recognize the significantfeatures of a well joint for visual guidance of a welding robot movingunder a taught path, and by interpolation to locate the joint root andits distance from the taught path for robot control and correction. Thistechnique, however, makes use of the direct knowledge of the taught pathof the robot or the part of the sensor system, and it fails to implementa feedforward approach.

Different look-ahead techniques have been used, in order to project thenext move of an effector end while computing and controlling inanticipation of where the effector end shall be. See for instance, U.S.Pat. Nos. 4,501,950; 4,542,279; 4,590,356; 4,663,726; and 4,675,502.However, none of these is teaching how to correlate in real time andunder feedforward control the sensed locations and the extrapolatedtaught path location in order to determine the required compensation inthe inherent robot manipulations along the sensed seam, or joint. Forinstance, U.S. Pat. Nos. 4,590,356; 4,542,279 and 4,501,950 essentiallyinvolve terminal homing, thus no static taught path is controlling. U.S.Pat. No. 4,663,726 is concerned with the robot system itself, i.e. therobot is not an a priori for the automatic seam tracker.

U.S. Pat. No. 4,675,502 involves a real time controller for a robotutilizing the robot taught path with a look-ahead vision system. Theconverted sensed locations data are stored in a queue, and a slightlyahead of the tool location is retrieved as a target, which is comparedwith the taught path for correction and control of the tool movementfrom present. No taught path recovery is taking place, and the elapseddistance, rather than the velocity, is taken into consideration forestablishing under feedforward control the anticipated tool position tobe aimed at with the robot.

The cross-referenced patent applications show the use of aninterpolation method to select a sensed location corresponding on thesensed path to the anticipated tool position for control. In thisregard, the prior art shows with U.S. Pat. No. 4,683,543 time-basedinterpolation control of a robot. However, in contrast with the presentinvention, it is not there to interpolate on the basis of elapseddistances. Another such time-based interpolation technique is shown inU.S. Pat. No. 4,663,726.

SUMMARY OF THE INVENTION

The present invention is used with an optical sensor of the look-aheadtype performing seam path tracking and applied to control of anindustrial robot done without the availability of the taught pathembodied within the robot system or without direct knowledge on the portof the sensor system of the taught path existing in the robotcontroller. A control processor is associated with the robot systemwhich, on the one hand, receives information regarding the effector endposition and orientation, with which an estimation of the taught path ismade while, based on previous control steps, the elapsed distance fromthe start of the tool on the path is constantly being determined,whereas, on the other hand, an image processor system is providing theelapsed distances from location to location as sensed, as well as thecorresponding-spatial coordinates for such locations with such data fromboth sides, the control processor iteratively extrapolates where theeffector end will be after at least one control step.

The present invention provides for finding the exact correspondingelapsed distance on the sensed path so as to retrieve the spatialcoordinates which are attached to it and it enables the controlprocessor in the same step to determine the error from a comparisonbetween the coordinates as sensed and the coordinates on the recoveredtaught path for the extrapolated, or anticipated effector end position.Control of the robot is effected in a feedforward fashion, control beingperformed with respect of the anticipated position while the effectorend is moving from the present position

The invention resides in establishing successive models of the seam pathbased on the sensed locations and in using such models toinstantaneously determine the anticipated location on the seam forcontrol by the control processor

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 represents in block diagram a robot system of the prior art;

FIGS. 2A, 2B and 2C show typical joints or seams with theircharacteristic profiles;

FIG. 3 is a schematic illustration of a taught path for a robot system;

FIG. 4 shows graphically the relation existing between the taught pathand the seam to be tracked by the tool of the robot in operation;

FIG. 5 is a representation of the parameters determining the positioningand orientation of the tool relative to the groove of a joint to beoperated upon;

FIG. 6 shows the axes involved in the orientation of the tool in worldcoordinates;

FIG. 7 is a block diagram illustrating control of a robot with a seamtracker using the feedback approach;

FIG. 8 illustrates in block diagram dynamic operation of a robot systemunder a stored taught path;

FIG. 9 shows an optical sensor using a light beam projection to form astripe on a joint and viewed by a camera for further treatment inascertaining the seam path in an auto-tracker;

FIG. 10 represents the optical system and defines the geometricparameters for stripe triangulation analysis;

FIG. 11 shows in block diagram the interfacing between the opticalsystem and the robot system, involving according to the invention animage processor and a control processor;

FIG. 12 is a block diagram showing how the control processor of FIG. 11effectuates feedforward control of the robot system;

FIGS. 13A-13D form a block diagram explaining the operation of thecontrol processor of FIGS. 11 and 12;

FIG. 14A is a graphic representation of the seam path and the recoveredtaught path, with the extrapolated tool location on the seam under athree-tick series of moves, as seen upon each cycle of the controlsequence of the industrial robot performed under the control processoraccording to the present invention;

FIG. 14B is a graphic representation of the locations in terms ofelapsed distance as sensed by the optical sensor along the seam orjoint, the sampled elapsed distance bracketing the elapsed distance onthe extrapolated location of FIG. 14A being identified, illustratively;

FIG. 15 is a block diagram representing in analog form the selection ofthe bracketing sensed samples of elapsed distance in response to theelapsed distance for the extrapolated position of the tool, and theconsecutive interpolation to derive tool coordinates as required forsuch position, and as used in FIG. 13C for the derivation of the errorand the generation of a control command for the industrial robot;

FIG. 16 shows illustratively how a seam path may be divided intosuccessive segments each approximating a linear function;

FIG. 17 is a block diagram illustrating the implementation of thefunctions of FIG. 16;

FIG. 18 is the block diagram of FIG. 13C as amended to perform modelingrather than interpolation, according to the present invention;

FIG. 19A, like FIG. 16 shows the seam path segmented according to themethod of FIG. 17, with correlation to the control operation under FIG.18;

FIG. 19B shows the successive models associated with the respectivezones of the segmented path of FIG. 19A;

FIG. 20 shows how, with a look-ahead sensor attached to a tool,orientation of the tool at a present position requires orientation toplace the sensor upon the seam;

FIG. 21 shows that effective sensing by the sensor requires the distanceof the sensor from the tool in the plane of the seam path to be a chordthereof;

FIG. 22 is a vectorial representation of the projection of the opticaldistance of the sensor to the tool on the seam plane defining with thechord of FIG. 21 the yaw angle to be compensated for;

FIG. 23 shows in block diagram the overall image processor used as seamtracker with a digital pipeline and its solid state devices monitoredand controlled by a microcomputer as used according to the imageprocessor invention to generate a pixel line representation of one edgeof the stripe image of the sensed seam and to derive seam pathcoordinates for the control processor (CP) to control the robot;

FIG. 24 is a more detailed view of the block diagram of FIG. 23;

FIG. 25 is a block diagram of the overall image processor (IP) showninterconnected with the control processor (CP) of FIGS. 13A to 13D;

FIG. 26A shows the ten possible situations characterizing a mask, orkernel, situated on the chosen lower edge of the stripe;

FIG. 26B shows illustratively three situations outside the tensituations of FIG. 26A;

FIG. 27 shows the pixel mask encoding for each of the thirteensituations of FIGS. 26A and 26B;

FIGS. 28A to 28D illustrate the steps of transforming the pipeline imagefrom 2-D to 1-D while reducing the pixel line from 2 to 1 pixel width,according to the image processor invention;

FIG. 28E illustrates the operation of a discrete second differenceoperator for corner detection according to FIG. 25;

FIG. 29 shows six illustrative representations of the shape of the seamcharacterized by the pixel line according to the image processorinvention;

FIG. 30 shows illustratively eight possible geometric relationships of acorner to its adjoining link, or links, as they characterize the shapeof the pixel line according to the image processor invention;

FIG. 31A shows a typical pixel line profile and FIG. 31B shows thecorresponding stripe image encoding;

FIG. 32A shows another typical pixel line profile and FIG. 32Billustrates the corresponding stripe image encoding;

FIG. 33 shows in relation to FIG. 31A, 31B and 32A, 32B the evolution ofa corner graph translating the shape of the pixel line in a particularcase;

FIGS. 34A and 34B give examples of extra corners identified on a pixelline;

FIG. 35 shows in five situations target refinement as made possible bythe image processor invention;

FIG. 36 is a diagram illustrating 1-D to 3-D image conversion;

FIG. 37 is a diagram illustrating image to slope conversion;

FIG. 38 is a flow chart illustrating the operation of the imageprocessor for pattern recognition, center point extraction andcoordinate determination for providing seam path location sample datafor the control processor of FIGS. 11, 12 and 13A to 13D.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring to FIG. 1, a robot system of the prior art is shown to includea robot processor RP combining with the taught path control signalderived on line 1 from a taught path pendant RTP the corrective inputderived on line 2 from a feedback signal indicative of an error in themoving and positioning of the tool manipulated by the robot RB. Thesignals of lines 1 and 2 are combined within the robot processor RP by acontrol summer interpreter which provides on line 3 the requiredparameters for control of the robot, namely regarding the tool 3-Dcoordinates x, y, z and the tool axes: lead/lag angle (or pitch), roll,and yaw. The parameters of line 3 are to this effect combinedmathematically in order to transform the world coordinates of theparameters of line 3 into tool coordinates on line 4. These areaccordingly controlling the servos which actuate the motors for lotionand the arms at the respective joints for positioning, so that the toolcan assume the required spatial position and orientation with respect tothe instantaneous seam path location of the robot.

As will be shown hereinafter, according to the present invention isprovided on line 1 (line 136 in FIG. 13A) a control signal obtainedunder a feedforward control approach by using (1) information regardingthe actual tool position and orientation as derived iteratively from therobot processor on line 5 (line 137 in FIG. 13A) and (2) informationsimultaneously and iteratively derived through an image processor froman optical sensor disposed ahead of the tool and optically tracking theseam, or joint, being worked upon by the tool.

Referring to FIGS. 2A, 2B and 2C, three types of joints, or seams, areillustratively shown under (a), as can be worked upon by the tool, awelding torch for instance, in order to join two pieces, or plates: P1,P2. Under (b) are shown graphically and viewed in cross-section, theoverall configuration of such a joint, typically a step, a V-shape, anda U-shape, respectively.

The invention will be described, illustratively in the context of awelding operation, with a torch positioned and set in motion along ajoint, or seam, to be filled-up with welding material into the groovethereof. Referring to FIG. 3, the robot is given a "taught path" tofollow, which allows the tool, or torch, to be positioned therealongwhile the tool is being controlled for operation in accordance with areference signal. Therefore, there is a basic correspondence between thetaught path and the seam path. The latter, however, is the exacttrajectory determining tool operation, welding operation. Accordingly,control of the tool into position for working upon the seam and of thecarriage motion along the taught path are related in space and time. Thefunctions of the signals of lines 1 and 2 in FIG. 1 are to insure suchspatial and time correlation when controlling by line 3 the motion andpositioning of the tool.

Referring to FIG. 4 which provides illustratively a representation of ataught path, the latter goes from an initial location L#1 to a secondlocation L#2 along a straight line. At location L#2 a 90° overallrotation is effected and motion is performed thereafter from locationL#2 to location L#3, also along a straight line but at right angle toline L#1-L#2. At location L#3 another rotation is effected by thecarriage of the robot under command of the taught pendant, and motion ispursued from location L#3 to location L#4, also along a straight line,in the example, but parallel to line L#1-L#2. The relation between thetaught path and the seam proper can be explained by considering only astraight line trajectory for the taught path like along L#1-L#2, orL#2-L#3, or L#3-L#4, in FIG. 3. To this effect, referring to FIG. 4, itis assumed that initially the robot is operative from an initiallocation ILTP on the taught path, while the torch, or tool, is initiallyat location ILSP on the seam path. It is also assumed, as generally donewith a robot system, that control is effected along the trajectory byelementary steps, or straight lines, or segments, not necessarily ofequal size, their duration being a time interval t, that is each isobtained under a control cycle as part of a sequence of elementarycontrol operations. As a result the robot will follow the taught path,from position ILTP successively past locations a, b, c, d, definingsegments ab, bc, cd, and the tool is being concurrently placedsuccessively at locations A, B, C, D, along the seam path, also withsegments AB, BC, and CD. In the course of the trajectory there is anelapsed distance counted from the starting point ED at location m forthe taught path, ED at location M for the seam path. These elapseddistances are being considered instead of elapsed times under thecycling process. Thus, the robot system will know at each time where therobot is on the taught path and, by a reverse process from tool control,it will also know where the tool is, hopefully, on the seam path.

The preceding, however, is an oversimplification. It is under theassumption of a perfect synchronism of the operations and of a trueparallelism of the trajectories. This is not the case in practice. Infact, as shown in FIG. 4, the seam trajectory differs somewhat from theideal. Moreover, the taught path is an oversimplified representation ofthe true seam, or joint, trajectory. Robot control requires precision inestablishing the relationship between the tool, or torch, and the seamupon which working is to be performed from point to point, orcontinuously. In other words, the tool should track the seam, that is,the robot should track the seam with the tool. The assumption is, then,that the seam tracker follows exactly the seam, or joint, and commandsthe tool to follow the seam as read. This is what is expected inprinciple from a feedback system, i.e. a system that can cancelinstantaneously any error detected and fed back from location tolocation. In the example of FIG. 4, a feedback system would insure thatthe deviations between seam path and taught path are instantaneouslycompensated for during the progression, thus, in accordance with thesensed values ea, eb, ec, ed, em from A to M on FIG. 4.

In reality, the interface between control of the tool under the taughtpath and control of the tool to keep track of the seam path is withinthe robot system very complex.

First, there is the nature of the weld joint, the nature of the tool, ortorch, and the nature of the operation to be performed with the tool,which affect how the tool should be moved and positioned. For seamwelding, for instance, a tool reference must be established taking intoaccount the arc, or gap to be maintained between the tip of the tool andthe bottom of the groove where welding material must be placed. There isalso the required voltage and current applied to the arc and the pitchof the tool held while advancing the welding wire, or presenting thetorch to the arc. Such requirements are illustrated in FIG. 5 where thetool is shown oriented in relation to the groove of the joint with atransverse, or bias offset d and a vertical offset, or gap h from thetip of the tool to the bottom of the groove.

There is another complexity, which is the ability of controlling a toolabout its six axes, namely the 3-D coordinates of the tool, plus thepitch, roll and yaw axes. Referring to FIG. 6, the tool is shownoriented along the a axis for the yaw, the o axis for the roll and the naxis for the pitch, while being positioned.

There is also the complexity in controlling a tool with a robot alongsix axes, namely the 3-D coordinates of the tool, plus the pitch, roll,and yaw axes. Referring to FIG. 6, the tool is to be oriented along thea axis for the yaw, the O axis for the roll and the n axis for thepitch, while being positioned at the tip of a vector p oriented withincoordinates x, y, z of a 3-D world coordinate system. Control willinvolve vector/matrix algebra, homogeneous coordinate transform theory,and the theory of discrete time linear dynamic systems in the process.

In position and orientation, the tool transform is given by _(w) T^(t)in world coordinates. The matrix involved is: ##EQU1## The vector p is aposition vector giving the coordinates of the tip of the tool expressedas _(w) T^(t) in 3-D robot base, or world coordinates. a is the"approach" vector (in the direction of applicator of the tool). n is the"normal" vector and o is the "orientation" vector.

From the _(w) T^(t) transform, is derived the inverse transform [_(t)T^(w) ]⁻¹ which is given by the relation: ##EQU2## Computation forcontrol by the robot system also involves differential transformation.Thus, a vector of increments in position and orientation is asix-element vector given as follows: ##EQU3##

where for the x-axis: Δ_(x) denotes a position increment in thex-direction and Δ_(x) is a rotational increment around the x-axis.Similarly, for the y-axis and the z-axis, a differential vector in onecoordinate system may be transformed into a differential vector inanother via the following equations:

Δx^(w) =δ^(t).(p×n)+d^(t).n

Δy^(w) =δ^(t).(p×o)+d^(t).o

Δz^(w) =δ^(t).(p×a)+d^(t).a

δ_(x) ^(w) =δ_(x) ^(t).n_(x) +δ_(y) ^(t).o_(x) +δ_(z) ^(t).a_(x)

δ_(y) ^(w) =δ_(x) ^(t).n_(y) +δ_(y) ^(t).o_(y) +δ_(z) ^(t).a_(y)

δ_(z) ^(w) =δ_(x) ^(t).n_(z) +δ_(y) ^(t).o_(z) +δ_(z) ^(t).a_(z).

The n, o, a and p vectors are drawn from [_(w) T^(t) ]⁻¹ in this case.

These elements of complexity are not encountered where robotdisplacement and seam trajectory involve a reduced number of degrees offreedom for the robot carriage and for the tool.

The present invention involves six degrees of freedom in toolpositioning and orientation. In addition, what has not been consideredin FIG. 3 is the complexity resulting from the fact that recognizingwhere the seam is actually located as sensed has to be coupled withwhere the tool has to be operatively disposed, and in real time for bothseam tracking and tool operation.

A straight forward approach to this problem has been the negativefeedback approach This approach is illustrated by FIG. 7.

In accordance with general feedback technique, the position of the toolis sensed by line 13, the seam is sensed by line 11. The detected signalof line 11 is amplified and scaled by an actual seam path sensing unitSPS so as to provide on line 12 a signal which is compared with the toolposition representative signal of line 13. Summer S1 provides an errorsignal on line 14 which is used by robot control to bring the error tozero. The tool reference signal is applied to summer S1 by line 10 sothat, in the absence of an error, the robot RB can position andorientate the tool as it should for operation on the groove, like inFIG. 5. Control is effected by the robot through a controller CNT, anarm ARM as generally known and as schematized in FIG. 1. The controlleris responding to the taught path from line 15 and to the control signalof line 14 which concurrently by summer S2 establish the control signalof line 16 into the controller.

Referring to FIG. 8, this portion of the controller operation is shownwith more details.

The taught path is derived from a list of locations LL defining thetaught path. Data from the list is dynamically translated into a controlsignal by a path planner PP. The controller is operated cyclically,typically with a period of 28 msec, so as to complete and settle thepreparatory steps for the next cycle. Tool positioning and seam locationdetermination, also typically, are effected on the basis of threeconsecutive locations, for averaging. This means that the values arestored for the present and the last two locations. When data for the newlocation is added data for the said location in the past is taken awayfrom storage. This is done for the control operation for each new toollocation. This is referred, hereinafter, as a 3-tick delay (3TD). Theerror of line 14 is thus, translated into an average error on line 14',which may be used cumulatively, or in a noncumulative mode, depending onthe position of a switch SW. In the cumulative mode, an adder receivesthe three-tick information and adds to it the feedback value from theoutput on line 18 which goes to summer S2. In the noncumulative mode,the error of line 14' via switch SW goes directly to summer S2.

The signal of line 16 within the controller CNT is transformed into itsvariable for the particular axis, or coordinate, for a total of sixjoints. FIG. 8 illustratively shows for one joint the individual errorsignal used by inverse transformation, at IKS, and the ensuing controlsignal of line 17 operating upon the joint angle under three feedbackloops, one for the torque (TFB), one for the velocity (VFB) and one forthe position (PFB), which are all concurring in bringing the error oncontrol line 16 to zero through joint adjustment, through a time-varyingjoint dynamic control loop JDL, via a control signal on output line 19to the joint motor.

Since the motion of the particular robot is controlled so that dynamiceffects of joint motion settle out within less than 28 milliseconds, itis only necessary to consider setpoint and time delay effects whenmodeling the robot for control purposes. The motion in the noncumulativemode may be expressed by

    .sub.w p.sup.t (k)=.sub.w p.sup.t.sbsb.taught (k)+p.sub.shift (k-3)

and

    .sub.w o.sup.t (k)=(k):Rot[x, δ.sub.x (k-3)]:Rot[y, δ.sub.y (k-3)]:Rct[z, δ.sub.z (k-3)]: .sub.w o.sup.t.sbsb.taught

where _(w) o^(t) (k) is the orientation submatrix of an hetm and Rot[x,δ_(x) ] denotes rotation around the x-axis by angle δ_(x). In thecumulative mode, the equations are: ##EQU4##

Feedback control requires sensing. Sensors had to be developed toprovide an informative system relative to the seam being tracked andgenerate data appropriate to the computational elements of a moderncomputerized robotic system. For this purpose, the most chosentechnology is electro optics. In these systems, for automatic seamtracking, a vision system is used capable of measuring the position andthe dimensions of the seam, or joint. This has been done, preferably,with a picture of a laser stripe reflected from the joint, as shown inFIG. 9. A light beam derived from a projector PRJ scans the joint JNTtransversely and a camera CAM captures at an angle φ to the plane of thescanning beam, the image of the stripe formed on the surface of thegroove and bordering planes or plates P1, P2. The image is liketypically shown under (b) in FIGS. 2A, 2B, 2C. The process is cyclicallydone, the camera image is read-out upon every frame, so that a strip isdetected in a discrete fashion at each of successive locations of astrobing sequence along the seam. A technique is used to determine thecenter C of the stripe which has to be followed along the seam from onelocation to the next.

FIG. 10 shows the optical system used to sense the seam along its trackand to derive a location-to-location tracing of the trajectory. Aprojector PRJ scans with a light beam LB the plate to be joined and hitsat a given instant, a point M thereon. The ray of light is defined byits angular position φ relative to the axis z of the optical system,namely the axis of the lens system symbolized by a lens LNS having acenter 0 and a plane orthogonal to axis z along an ordinate y. Behindthe lens, at a distance f from the center is the focus plane whichcarries the image array IA upon which for each impact M of theprojector's beam, corresponds an image point m. M has an ordinate y andan abscissa z, while m, has an ordinate j and an ordinate i, the latternormal to the plane of FIG. 10 and in the plane of array IA. The opticalsystem has 3-D coordinates x, y, z, the first x being normal to theplane of FIG. 10 at center 0. The image of the strip containing point mis a 2-D image in the plane of array IA of coordinates (i, j). If in theplane of FIG. 10, axis y intersects the beam of light at a distance b,and the axis z of the optical system is at an angle φ to line mM passingthrough center 0, basic stripe triangulation analysis leads to thefollowing equations:

    y=tan [φ].[z]-b                                        (1)

    y=tan [φ]. z                                           (2)

Using similar triangles, tang φ can be related to image coordinates.Thus, if J_(m) in the maximum vertical resolution of the sensor andk_(j) the vertical distance between pixels on the camera array IA, jbeing the ordinate of m on the array, the light stripe image on thesensor from the center point C of the image (i, j) is: ##EQU5##

k_(j) is the vertical interpixel distance

J_(m) is the maximum vertical resolution of the sensor

The value of tan φ is then ##EQU6##

where f is the distance from the lens center 0 to the focal plane. (Thisdistance is not the focal length of the lens, unless the lens is focusedon infinity.) Substituting into equation 2: ##EQU7## Solving the last yequations simultaneously: ##EQU8## by substituting z into equation (1):##EQU9## where k_(j) is the vertical interpixel distance and J_(m) themaximum vertical resolution of the sensor.

By a similar analysis for the x-direction, ##EQU10## where

k_(i) is the horizontal interpixel distance, and

I_(m) is the maximum horizontal resolution of the sensor.

Equations (3) and (4) are functions of constants which are known orcalibrated and a single variable, j. The dynamic range of j is notexcessive (only 1 to 512) so that those equations may be implemented aslookup tables. Equation (5) is a function of z constants and the imagevariable i. It may also be implemented in a rapid fashion using a lookuptable and one multiplication.

One of the three rotational degrees of freedom can be obtained fromanalysis of the slope in the image. The surface slope component m_(zx),defined by ##EQU11## is related to the image slope Δj/Δi, when thestripe projector is aligned to put lines parallel to the i axis when aflat surface is normal to the lens principal axis. The relationship isnot simple, nor linear, but may be derived through the use of totaldifferentials. For instance, for dZ: ##EQU12##

Referring to FIG. 11, the optical system provides on the camera an imagein two dimensions (i, j) like shown under (b) in FIGS. 2A, 2B, 2C. Byline 80, a digitized information as a video camera derived signal, isinputted into an image processor IP. Within the image processor IP atCD, the center C of the seam profile is determined by its coordinates i,j. By line 81 such information is passed to a processor sub-unit SLCwhich collects the successive images along the x axis, and provides theseam path in 3-D coordinates. Accordingly, on line 82 is derived, as thelocus of the successive locations of the center C, the seam path in 3-Dcoordinates, thus, as sensed optically.

On the other hand, the robot processor RP knows where the tip of thetool in world coordinates has been positioned at a given instant. Thisis expressed as _(w) T^(t) and derived on line 137 from the robot andinputted in the control processor CP. The control processor CP passesthis information to the image processor IP on line 137'. On the otherhand, the control processor sends to the robot processor a correctivesignal on line 135 to compensate for an error in tracking the seam path.At this point, it is observed that the optical sensor being mountedahead of the tool, the sensed location for the seam track, on line 82,is not where the tool is. The tool will reach the sensed location aftera certain delay in processing cycles, typically 100 ticks, which with a28 msec/tick amounts to 2.8 sec.

As explained hereinafter, according to the present invention, the sensedlocations are by the image processor identified successively for eachlocation while indicating how far from the start, i.e., the elapseddistance on the seam, and the corresponding 3-D coordinates and 3-axisorientation for the tool if it were there. These values are stored intothe control processor by line 103. The elapsed distance is provided bythe control processor CP to the image processor IP via line 169 in orderto correlate it with the elapsed distance of the tool, the latter beingbased on information derived from the robot processor RB by line 137,which is the tool actual tip coordinates in position and orientation atthe present cycle of control. At this time, the control processorextrapolates from the last two locations where the tool will be assuminga linear progression along the seam path. At the same time, the controlprocessor recovers the taught path which is not directly available fromthe industrial robot and also extrapolates on the recovered taught pathfrom the two last locations for which the two last controls have beenexercised. Using the elapsed distance for the tool, the informationreceived on line 103 from the image processor IP as to elapsed distancesand correlated sensed locations in spatial position and orientation, theactual elapsed distance corresponding to such extrapolated position ofthe tool is determined. One approach to do this is by interpolationbetween two consecutive sensed locations and from it the correspondingactual anticipation for the tool is identified in terms of position andorientation.

From the latter interpolation results and from the earlier extrapolationposition and orientation an error is established by the controlprocessor which is used by line 136 to control the robot in relation tothe taught path, thus, within the robot processor RB. It has beenassumed earlier that a three-tick spread is performed in the controlprocess. This means that the extrapolated location for the tool islooked upon at instant k+3 ; where k is the present instant of controland calculations. Similarly, recovery of the taught path is effected atinstant k based on a controlling error involved in control of the robotat instant k-3.

Referring to FIG. 12, feed-forward control according to the invention isillustrated in block diagram. The optical sensor provides for the imageprocessor IP all the information necessary to determine where the centerC of the strip is located, in real time for each location in thesequence of images with the camera CAM. Accordingly, at the output ofthe image processor IP is known the actual location of the seam, namelyon line 103, or 121. By line 137 the robot RB knows where the tool tipis located in world coordinates. From such tool location, which isbehind the series of seam locations detected by the sensor, block BLKeffectuates an extrapolation of the tool position. It is assumedhereinafter, that the time difference is of 3-ticks, which means that ifthe tool at instant K (present time) is as shown by line 137, it willtake 3 cycles to effectuate the full correction at the extrapolatedlocation when it is needed. Therefore, the tool tip position _(w) T^(t)is extrapolated to instant (k+3). Within block BLK2, summer S1 compareswhere the corresponding seam location is, as known from lines 103, 121,and an error is derived as required to adjust normal control, inresponse to line 126 under the tool reference, thereby to provide acorrected input into the robot RB, by line 136. Otherwise, the robotfunction is like under the feedback approach of FIGS. 7 and 8, toposition and orientate the tool so that the tool tip be properlypositioned regarding the seam as sensed by the optical sensor.

First the internal organization of the control processor CP will beconsidered by reference to FIGS. 13A-13D, 14A, 14B and 15; then,consideration will be given to the structure and operation of the imageprocessor IP.

Referring to FIG. 13A, the control processor is interposed between therobot processor RB, with which it communicates through robot ALTER port102, and the image processor through IP port 101. The industrial robotcontrols the effector end in position and orientation in relation to ataught path, as generally known. The taught path, however is not knownoutside the robot. According to the present invention the taught path is"recovered" by the control processor for use therein. To this effect, atthe present instant k, information is derived on line 137 from robotALTER port 102 as to where the tip of the tool actually is at suchinstant. This information is represented by transform _(w) T^(t) (k) inworld coordinates. As earlier stated the matrix involved is' ##EQU13##where the vector p position vector gives the coordinates of the tip ofthe tool, and where the projections at right angle to one another of thea (approach), n (normal) and o (orientation) vectors are given as shown.This is tool position D on the seam path for instant k as shown in FIG.14A. At instant (k-1) the position was at C. Since control is effected(illustratively) on a three-tick step, the tool position D on the seampath at instant k, results from a correction for a required deviationD(k-3) from the taught path location dd, determined and commanded fromthe robot three cycles Δt earlier. The taught path not being known fromthe industrial robot, the information on line 137 is used to "recover"the taught path as follows:

Referring again to FIG. 13, by line 140 the position vector p is derivedand by line 152 the deviation D(k-3) is retrieved from a queue 151 whereit had been stored. Lines 140 and 152 are feeding information into block143 where is computed

    .sub.w p.sup.t (k)-Δp(-3)

thereby providing point d of FIG. 14A in term of spatial position.Similarly, from lines 140 and 152, block 144 provides the orientation(δx, δy, δz) for the estimated taught path position d by using _(w)T^(t) :Rot⁻¹. This being done for all past instants (k-1), (k-2), (k-3),. . . the taught path is recovered as segments, illustratively shown forthe last locations a,b,c,d. As a matter of fact the "recovered" path wasinitiated at the origin O of the joint. Thus, on line 153 is obtainedthe transform _(w) T^(t) _(taught) (k) for instant k and, consideringblock delay 154, on line 155 is obtained the earlier position on therecovered taught path _(w) T^(t) _(taught) (k-1).

At the same time (FIG. 13B) on line 138, like on line 139, is derivedfrom line 137 the vector position _(w) p^(t) (k) for instant k, and online 139 with a delay of one cycle is now derived the vector position_(w) p^(t) (k-1), which had been stored in block 142. Block 141 respondsto line 138 and to block 142 to provide on line 161 ΔS_(t) (k) accordingto the equation:

    ||p(k)-p(k-1)||=ΔS.sub.t (k)

This is the spacing between points C and D on the seam path graph ofFIG. 14A. This being done successively from the start, (still in FIG.13B) the segments are totalized by blocks 163 and 165 (where block 163adds up the last segment to the previous total of line 168), therebyproviding on line 170 the "elapsed distance" since the start on the seampath for the tool, namely: s_(t) (k). At the last cycle was delayed theresult of line 170, so as to make available at instant k the lastelapsed distance, thus, the total until point C for instant (k-1).Therefore, the two elapsed distances appear on respective lines 176 and177 and are to be applied both to blocks 173 and 175, as explainedhereinafter. Similarly, from line 161 and by blocks 162 and 164, isdetermined the elapsed distance s_(s) (k) for the sensor on the seampath, as counted from the origin O, but with a phase difference relativeto the tool as shown by the distance--s_(t) to the virtual initial startO' of the tool behind the sensor origin O. The current elapsed distancefor the sensor s_(s) (k) is sent to the image processor by line 169 andthrough image processor port 101. As earlier stated, the sensor is phaseshifted ahead of the tool by, typically, 100 msec. compared with controlthe cycle of Δt=28 msec., also typically. As explained later, the imageprocessor IP is associating each of the sensed locations with adetermined tool spatial position and orientation. Thus, for each elapseddistance sampled for such location (s_(s) (k)) there is a correspondingworld coordinate transform _(w) T^(S) (k). Referring to FIG. 13C, thesepaired data, received through the IP port 101 are stored into a queue104 from the start and classified by consecutive elapsed distances (usedas an address in the RAM chosen for such purpose). By looking at instantk to a particular tool position anticipated by the control processor, itwill be determined with the corresponding IP elapsed distance (oraddress of RAM 104) what the sensed coordinates and orientations are forthe tool at such location (line 125). The recovered taught pathcorresponding location (line 128) will permit the determination of thedeviation (D(k) for instant k to be used in controlling the robot byline 126 in bringing back the tool on such sensed location.

As shown in FIG. 14A, the tool path anticipated tool position isobtained by extrapolation linearly from segment CD, which leads toextrapolated position Ex. C and D are given in terms of elapsed distanceby blocks 171 and 165. To extrapolate tool elapsed distance in threeticks, the tool path extrapolated tool position is at instant (k+3) atan anticipated distance given by the formula: ,mod₋₋ equ 1 ##EQU14##wherein t(k)-t(k-1) is equal to Δt, equal to 28 msec. Therefore theformula reduces itself to:

    S.sub.t St(k+3)=S.sub.t (k)+3[S.sub.t (k)-S.sub.t (k-1) ]

This equation is a simple velocity based predictor. This function isperformed within block 173 of FIG. 13B in response to line 172 and block171. Therefore, on line 174 is outputted s_(t) (k+3). This anticipatedelapsed distance, known on the seam does not reveal though where thelocation actually spatially is. To find out, it is first determined byextrapolation on the recovered taught path where the extrapolatedcorresponding location (FIG. 14A) is located. This is done by relatingthe transforms of points c, d, and e, on the recovered taught path tothe elapsed distances for C, D and Ex on the seam path (FIG. 14A). Thisextrapolation on the taught path obtained from the samples of therecovered taught path is given by the equation:

    .sub.w T.sup.t.sub.taught : {f.sub.e w T.sup.t.sub.taught : [.sub.W T.sup.t (k-1)].sup.-1 } in which f.sub.e is the coefficient of extrapolation given by the formula: ##EQU15##

The extrapolation coefficient supplied by block 175 (FIG. 13B) inresponse to lines 176 and 177, the output being on line 178. Block 157yields, according to the above equation, the extrapolated position (online 128) from a calculation involving lines 155, 156 and 178. Knowingwhere e_(x) on the recovered taught path (FIG. 14A) is located, itremains to look among the samples sensed by the image processor for onematching the same elapsed distance as for the extrapolated tool positionat Ex (FIG. 14A). The true sample will reveal from the queue 104 whatthe position and orientation coordinates are for that particularlocation and, therefore, where the true extrapolated position for thetool E should be, rather than at Ex (FIG. 14A). Such knowledge will alsoenable to determine the deviation D(k) with which the control processorcan bring about, through control of the robot, the right toolpositioning and orientation, i.e. on the seam path.

In order to find which sample matches the elapsed distance s_(t) (k+3)derived on line 174 for point Ex of FIG. 14A, two approaches arehereinafter described. One is by interpolation, as shown in FIG. 13C.This is described in U.S. Pat. No. 4,813,381 entitled "Optical Trackerand Real Time Control System for an Industrial Robot". Another approachis by modeling. It is the object of the present invention. Theinterpolation method will be described first, by reference to FIG. 13C.To this effect, the afore-stated elapsed distance for point Ex of FIG.14A, derived on line 174, is used to interpolate between consecutivesample values of the queue 104. Those two consecutive retrieved valuesare indicated as s(l) and s(l-1) in block 180 of FIG. 13C. How these twovalues are found and retrieved is illustrated by the block diagram ofFIG. 15, although, it is understood that the functions therein areimplemented in software A scanner SCN scans the addresses of the RAMwithin block 104 in the order of the data list. Each sample derivedduring scanning from line 25 is passed on lines 26 and 27 to acomparator CMP1 and on lines 26 and 28 to a comparator CMP2. ComparatorCMP1 receives as a threshold a reference signal representative of thevalue S_(t) (k+3) of line 123. When the value scanned and passed ontoline 27 becomes smaller than the value of line 123, comparator CMP1generates on line 29 a control signal for a latch LTC1 which retains atthis instant both the value of the sample and its associated transform,thus, s_(s) (l) and _(w) T^(S) (l). Similarly, comparator CMP2 respondsto the signal of line 123 becoming larger than the sample scannedthrough by scanner SCN and causes, by line 31, another latch LTC2 tolatch the values from lines 25 and 32 at such instant. Accordingly,latches LTC1 and LTC2 provide on lines 121 into block 122 of FIG. 13Cthe two samples bracketing the value of line 123. Block 122 is aninterpolator responding to an interpolator factor f_(i) defined by block180 in accordance with the signals of lines 121 and 123. It appears thatthe block diagram of FIG. 15 is part of block BLK2 within the controlprocessor CP of FIG. 13C. The elapsed distance for the tool (at Ex inFIG. 14A and given by line 123) is used between s(l) and s(l-1) derivedfrom lines 121 to establish within block 180 the interpolation factorf_(i), which is outputted on line 124 for block 122, fi being given by:##EQU16##

With such interpolation factor, the transform values associated with thesamples s(l) and s(l-4) are interpolated by the equation:

    .sub.w T.sup.S (k+3)=.sub.w T.sup.S (l): {f.sub.i ·.sub.w T.sup.S (l): [.sub.w T.sup.S (l-1)].sup.-1 }

In practice it has been proven more time efficient to linearlyinterpolate each element of the n, o, a, p vectors from the transforms.Therefore, for the n component for instance, and the p vector, theequation becomes:

    .sub.w p.sup.S (k+3)=.sub.w p.sub.x.sup.S (l-1)+f.sub.i [.sub.w p.sub.x.sup.S (l-1)-.sub.w p.sub.x.sup.S (l)].sup.-1

There is a total of twelve equations of the same form as above.

Accordingly, block 122 effectuates the calculation and the result is_(w) T^(S) (k+3) appearing on line 125, which represents the truespatial position and orientation of the tool on the seam asextrapolated, namely as anticipated (E on FIG. 14A).

According to the present invention, the deviation D(k), which at instantk would exist between the recovered taught path position e and theanticipated tool position E (FIG. 14A), is obtained on line 129 withoutusing the interpolation method illustrated by blocks 122 and 180 of FIG.13C. Instead of retrieving two consecutive elapsed distance sampless_(s) (l) and s_(s) (l-1) encompassing the elapsed distance value s_(t)(k+3) of line 123, it is now proposed, as an alternative solution, tostore series of successive samples collected well ahead of the arrivalof the tool, to analyze the nature of such succession of samples and togenerate a model which is a function of s providing a representation ofsuch series of samples. Then, in a second stage, concerned with theeffective arrival of the tool within the actual portion of the seam pathrepresented by such series of samples earlier stored and modelled, themodel is used to deliver the exact tool position and orientationmatching the anticipated elapsed distance aimed at.

Referring to FIG. 16, from A to B along the seam path as sensed and asstored into the queue 104 of FIG. 13B, a series of elapsed distances, assampled and stored in the queue, appear to remain substantially within asegment AB of definite slope. Then, at point B, there is a substantialchange in the slope, and the path takes a new orientation. From B to C,such new slope maintains itself, that is, until at C a new orientationtakes place along CD. The first step, according to the invention,consists in collecting the samples and classifying them according to acommon geometrical feature, or function, thereby performing asegmentation of the series of samples within queue 104. Since the sensoris working a relatively remote distance from where the tool is found inmotion on a three-tick control progression, a zone is established withinqueue 104 ahead of the tool and extending up to the actual sensinglocation at instant k. Thus, if k is the instant the tool is to becontrolled for an extrapolated position (k+3), the most remote sensedlocation on the seam path at instant k is located at (k-n) ahead of thetool, and the first sample to be collected in the segmenting processjust mentioned would be at (k-n+m), thus the closest from where the toolis. Having detected at instant t_(A) (FIG. 16) a change of slope andidentified a new segment of characteristic slope, and so on at instantt_(B), t_(C), . . . groups of samples are separately storedcorresponding to such successive segments, defining zones in the seriesof samples from the queue, like s_(A) -s_(B), s_(B) -s_(C), s_(C)-s_(D), . . . in FIG. 16. Such segments, or collections of data forsuccessive locations, are each defined as a model by the computer. Forinstance, the first and the last sample in the segment, or group ofsamples, are chosen as the two ends of a line, and the function is sodefined. Another solution is to use the linear regression method leadingto the same result.

Path segmentation is accomplished under several rules applied to thesamples:

1. No segment shall be longer than L so many points, i e so manysamples, typically L=20;

2. a segment boundary shall occure where a significant corner occurs inany of the parameters implied by the sample, namely p_(x), p_(y), p_(z),n_(x), n_(y), n_(z), o_(x), o_(y), o_(z) (it being reminded that fora_(x), a_(y), a_(z), the approach vector is redundant and follows fromthe vector cross product n=o×a) as expressed as functions of the elapseddistance If T is the limit characterizing a corner, a corner detectionfilter will be defined as follows: ##EQU17## where f is the function ofK rank K and where T is the chosen sensitivity of the filter.

Accordingly, corners in position (p_(x), p_(y), p_(z)) or changes inorientation (n_(x), n_(y), n_(z), o_(x) o_(y), o_(z)) will cause theformation of a new segment;

3. If a segment has an Euclidean length which is less than a limit Lmin,the vector o parameter (which is the one characterizing the progressionforward along the path) will remain the same as for the precedingsegment, thereby lengthening the segment under the same slope;

4. There will be a new segment boundary whenever upon three consecutivepoints the action code a(k) is such that a(k+1)=a(k), and a(k)≠a(k-1);

5. No segment will have less than two seam points;

6. Adjacent segments will have common endpoints in all DOF.

These rules will insure that segments are provided which are connectedwith no overlapping and all bounded in length.

Having so determined a series of segments with their boundaries, thesystem is able to establish for each segment a parametric model. Suchsegment model includes the path (defined by vector p) and theorientation (given by n, o, a) and an action code. Each element of the pvector is modeled as an affine linear function of the elapsed distances. Therefore, as illustrated in FIG. 16, segments AB, BC, CD, arerepresented parametrically by equations like

f(s)=m.s+b

where f is any of the position vectorial components p_(x) p_(y), p_(z) ;where m is the slope and b the intercept. The latter are computedseparately for each element of p by one of the two methods:

the two endpoints are used to determine the line passing therethrough;or

the least-squared error line is sought which fits all the points of theparticular segment.

The orientation vector is constant for the segment. It is determined asthe scaled unit vector u_(n) which from the initial seam sample of thesegment is pointing toward the final point of the segment. The normalvector n (which is orthogonal to the o vector) is modeled in the sameaffine linear fashion as the p vector. Thus, there are three linearequations to be computed as a function of the elapsed distance (the avector, as earlier stated being obtained from a computation of the crossproduct between the n and vectors). The parametric segment transform isas follows: ##EQU18## where s stands for ^(s) start<s<^(s) send. Theseam path is computed from such parametric transform for any value of swithin the segment bounds. The result is a smooth seam path interpolatorwhich, as distinguished from the embodiment of FIGS. 13B and 13C, doesnot require the filter BLK3 of FIG. 13C.

It is observed that with this approach the vector is derived bycomputation only at the time the seam path is being segmented.Therefore, it is not obtained from the image processor IP which feedsinto queue 104 through port 102. In other words, the image processor,besides the position parameters, provides only information aboutorientation around the y-axis. Therefore, the original n vector from thequeue 104 is not necessarily orthogonal to the vector. This situationrequires to distinguish a component which is orthogonal to the vector oand one which is not Using the same technique as shown in FIG. 18 forvectors MR1 and R10, the non-orthogonal component is removed byperforming the vectorial difference:

    n orthog=n-n.o

where n is the original seam sample parameter derived from the queue 104and where o is the orientation vector computed at the time the seamsegment is being modeled. Accordingly, the parametric equations for theparticular segment model are computed using in the matrix n orthog,rather than n.

Referring again to FIG. 16, it is observed that while the functions usedin the models along the path shown illustratively are all linear for anumber of intermediary samples, if at a point of detection of a changeof orientation leading to a new segmentation and modeling action, asuccession of such changes would occur at regular intervals, the pathwill in fact follow a contour which is circular. In such case, the rulesestablished hereabove illustratively, would bring a change oforientation upon each consecutive sample derived from queue 104.

Referring to FIG. 17, a block diagram illustrates how the afore-statedfunctions are implemented. A fast scan circuit by line 51 scans queue104 of FIG. 13C recurrently between the last location sensed by thesensor, namely at instant (k-n), and an earlier location, namely atinstant (k-n+m) which is far enough from the present operative zone ofoperation of the tool (namely for control ranging from instant k toinstant k+3). While scanning quickly, the samples are retrieved by lines51 and 52 and inputted into a slope detector. This means that for everytwo successive sample positions the coordinates are compared and theslope therebetween determined, by detector SLD. The outputted slopesignal is, by line 53, inputted into a slope derivative detector SDD.The outputted value, on line 54, is inputted into a comparator CMP3having a threshold established by a reference signal appearing on line55. Data retrieved from lines 51 and 52 are stored into a correspondingone of a number of available storage circuits, via lines 61. Theparticular storage circuits enabled to store by a sequencer SQA which istriggered into position by a set signal from line 56 whenever comparatorCMP3 has had its threshold exceeded by the change of sloperepresentative signal of line 54. At the same time the comparator isreset by kine 57 for subsequent operation. It is observed that, byreference to FIG. 16, whenever there has been a change of slope at pointA, translated by a signal on line 54 exceeding the reference of line 55,sequencer, for instance, is set to position #1 corresponding to store #1by line 58, and the data from line 61, namely for the zone #1 within 104will be stored therein. At the next change of slope, namely at B in FIG.16, sequencer will be moved into position #2 whereby line 58 enablesstore #2 and storing therein of data from zone #2 in queue 104, and soon for the successive zones of 104 like ab, bc, cd, and de, with storagecircuits store #1, store #2, store #3 and store #4, respectively.Indeed, there are as many storage circuits as there are changes oforientation for the sensed path which are detected by comparator CMP3,and so many corresponding positions for the sequencer SQA.

This has taken place very quickly by scanning with the fast scan circuitFSC. The computer is able to handle rapidly the information within thesuccessive modeling circuits MD1, MD2, MD3 and MD4, in response tocorresponding lines 62 providing the stored data for modeling. Asexplained earlier, in response to data of store #1, an equation like f(s)=m.s+b is established within modeling circuit MD #1. It is assumedthat tool motion represented by the tool tip coordinate of line 137(FIG. 13B) is being detected by zone detector ZD as entering zone #1,namely through line a at instant (k-n+m) becoming instant k, wherebyanother sequencer SQB is set into its position #1, for instance, by aset signal on line 63, thus at point A of the earlier sensed path, nowfor the tool. This means that Model #1 is activated by line 65 so thatwhenever the sample of line 123 (extrapolated position at E in FIG. 14A)for such value of the elapsed distance by line 125 is derived the valuegiven by the model, thus, the position and orientation of the tool forposition E which match exactly the anticipated value of the elapseddistance. This operation is repeated for the next zone (zone #2) thedetector having been reset by line 64 following triggering of sequencerSQB into its next position. It is seen that FIG. 17 includes twoportions operating in two different stages, one being preparatory to theother. At the earlier stage, segmentation is effected by comparator CMP3and sequencer SQA, based on data from the queue 104, and during the samestage modeling is performed within modeling circuit MD1, MD2, MD3, orMD4. The second stage goes with the motion of the tool and the controlof the robot processor RB. Line 123 is applied to the correspondingmodel and the solution to the interpolation is derived, by line 125,from a corresponding modeling circuit

FIG. 18 shows how the block diagram of FIG. 13C is modified in order toimplement this embodiment of the invention. The block diagram of FIG. 17has been shown there as a block BLK6 which responds to the signal ofline 123 and outputs for block BLK2 the signal of line 125, the latterresponding to the modeling circuit MDL treating data stored in thestorage circuit STR according to the scanned values from 104 by 190.Otherwise, FIG. 18 is like FIG. 13C. FIG. 19A is a repeat of FIG. 16showing graphically the relationship between the zones of queue 104 andthe functions within the modeling unit (Model 1, for instance, containsa function fl(s) and outputs on line 125 the solution _(w) T^(t) (k+3)for the interpolation of the variable s_(t) (k+3) on line 123. FIG. 19Bis representing how block MDL of FIG. 18 is relating the models (MODEL#1, . . . ) to their respectively associated zones (ZONE #1, . . . ) inthe segmented seam path.

Block 127 of FIG. 13C now compares the coordinates on the recoveredtaught path (line 128) with the just found coordinates for the toolposition and orientation on the sensed path in relation to the elapseddistance S_(t) (k+3) anticipated at instant (k+3). Block 127 also takesinto account the reference signal generated by block 126 which defineswhat the tool operation should be with regard to the seam (in welding:the offset, the gap, the voltage and current, . . . ). The generalequation applied by block 127 is expressed as follows:

    .sub.w.sup.err T.sup.w (k+3)=.sub.w T.sup.S (k+3): [.sub.w T.sup.t (k+3).sup.Ref : .sub.t T.sup.S ].sup.-1

This is the function performed by summer S1 in the feedforward controldiagram of FIG. 12. The equation:

    hd w.sup.err T.sup.w (k)=.sup.act.sub.w T.sup.S (k): [.sup.Ref.sub.w T.sup.S [.sup.-1

leads to the differential vector necessary for path modification byextracting therefrom:

D(k)=[^(err) _(w) P_(x) ^(w), ^(err) _(w) P_(y) ^(w), ^(err) _(w) K_(z)^(w), ^(err) _(w) δ_(x) ^(w), ^(err) _(w) δ_(y) ^(w), ^(err) _(w) δ_(z)^(w) ]^(T)

It is observed that the basic feedforward error equation contains fivenon-zero elements which are: the three position errors, plus the rollerror, which happens to be a rotation about the y-axis, and the pitcherror, which happens to be a rotation about the x-axis, the latter beingtwo of the three tool coordinates. As explained in U.S. Pat. No.4,843,284 for "Path Contriving System for an Optical Seam Tracker", andas described hereinafter, the third tool coordinate, namely the yawangle, is controlled according to a different criterion and controlrule. The "preferred" tool position on the seam with respect to therobot world coordinates is given by:

    pref.sub.w T.sup.S (k+3)=.sub.w T.sup.t (k+3): .sup.Ref.sub.T.sub.t T.sup.S

It is the position the seam should normally be in order to maintain thetool under the offsets required by the reference signal of line 126 inFIG. 12, namely: ^(Ref) _(t) T^(s).

Actually, as known from the interpolator of FIG. 15, and as derived online 125, the seam position is according to:

    .sub.w T.sup.S (k-l+3)=.sub.w T.sup.S (k-l-1): {f.sub.i w T.sup.S (k-l-1): [.sub.w T.sup.S (k-l)].sup.-1 }

which, with the equation error:

    .sup.err.sub.w T.sup.w (k)=.sup.act.sub.w T.sup.S (k): [.sup.Ref.sub.w T.sup.S ].sup.-1

leads to the error transform:

    .sup.err.sub.w T.sup.w (k+3)=.sub.w T.sup.s (k+3): [.sub.w T.sup.t (k+3): .sup.Ref.sub.w T.sup.s ].sup.-1

From the latter is extracted the positional error vector ^(err) p^(w)(k+3) directly as the p vector from w^(err) _(w) T^(w) (k+3). As to therotational error vector, it is extracted by recognizing the form of theorientation submatrix for small angle deviations to be: ##EQU19##

Therefore, the differential motion vector (without the yaw orientation)is: ##EQU20##

This amounts to establishing the deviation D(k) which at instant k wouldexist between the recovered taught path position e and the anticipatedtool position E, independently of the yaw error, as earlier stated andas explained hereinafter. This is an error to be compensated for bycontrol via line 136 (FIG. 13A) through the ALTER port 102 of the robot.The deviation D(k) represents ΔPx, ΔPy, ΔPz, for the 3-D coordinates andΔθ1, δθ1, δψ1 for the three orientations of the effector end. Theyappear on line 129 at the output of block 127 (FIG. 13C). To such errorsare added and distributed (again, as explained later) the coordinatevalues δ_(x) =a_(x) Δψ, δ_(y) =a_(y) Δψ, and δ_(z) =a_(z) Δψ,representing the corrections from the yaw chord calculator of block BLK1(FIG. 13D) derived on line 131. In block 130 the values of lines 129 and131 are composed to provide on line 132 the values Δx, Δy, Δz, Δθ, ΔφΔψ,which are passed by line 133 through a filter shown within block BLK3and yielding at the output the control signal of line 136 which, besidesthe ALTER port 102, is also passed by line 150 into the queue 151maintaining the three last corrections stored.

As shown in FIG. 14A, the robot is moving the tool along straight linesupon each control step, like segments AB, BC, CD. From one segment tothe next, it has been shown how the position and orientation of theeffector end at instant k has been determined from the last move, and isused for the next move (see for instance block BLK2 within controlprocessor CP of FIG. 13F) Since such iterative progression step by stepof the tool is based on a plurality of known elapsed distances andposition/orientation transforms for the seam path as sensed (see forinstance block 104 in FIG. 13C), it is necessary that the look-aheadsensor be positioned above the seam at all time, so many ticks ahead ofthe tool, 100 ticks in the illustration given. This is not a problemwith a strictly linear joint, since the sensor will move in the sameplane with the tool. If, however, the seam follows a curve, there willbe no longer any alignment and, as shown in FIG. 20, the sensor willremain along the tangent of the seam in position PS1, whereas foroptical sensing it should be in position PS2 above the seam. As shown inFIG. 21 in the plane of the seam, when the seam has the right positionPS2, the projection MC of the vector joining the tip of the tool to thecenter of the camera is a chord between the operative point M of thetool on the seam at instant k, and the location C where the seam issampled optically. Therefore, a rotation by Δψ is required to bring thesensor from position PS1 to the desired position PS2 (FIG. 20).

Referring to FIG. 22, the relation between the optical system and theplane of the seam is illustrated at the time at point M the tool hasbeen positioned and oriented in accordance with the given of block 127and line 129 of FIG. 13C. In other words, tool TL has been positioned ndoriented according to ΔPx, ΔPy, ΔPz, Δθ1, Δφ1, Δψ1. At this time, thesensor SNS (optical center 0) which is in a fixed relation with respectto the tip of the tool (but does not have to be, depending upon thechosen installation) projects itself onto the seam path plane at R1,whereas the desired position would be at R2 on the seam path, at the endof the chord MR2 The angle to rotate MR1 into position MR2 is Δψ. BlockBLK1 of FIG. 13D illustrates how the angle Δψ is determined, and how theso derived angle is, by line 119, block 120 and line 131, combinedwithin block 130 with the data of line 129, so as to rotate the tool anddisplace the sensor from R1 to R2 (FIG. 22).

Recourse must be had to vector/matrix algebra and homogeneous coordinatetransform theory. Vectors are denoted with lower case underscoredletters, like (n). Vector components are denoted with a subscriptedlower case letter, like (n_(x)) for the x-axis Homogeneous coordinatetransform matrices will have a sub- and superscripted upper case letterT, like (_(w) T^(S)). Accordingly, at point M the position andorientation of the tip of the tool along MR1 is given by the matrix:##EQU21##

Vector p is a position vector represented by _(w) T^(t) in the worldcoordinates. There are three other vectors mutually orthogonal atlocation M: n (for normal), o (for orientation) and (for approach), asearlier stated herein. Each vector is a unit vector. They are related bythe vector cross product as n=oxa. Referring again to FIG. 22, knowingMR1 and MR2 the angle inbetween can be determined. MR1 being the chord,is known from the last sample obtained with the sensor ahead of thetool. Assuming n is the number of ticks separating the tool from thesensor (as earlier stated 100 tick would be typical), at instant k (forposition M of the tool) the sensor is at (k-n). At the bottom of thequeue within block 104 is the transform for point R2 on the seam. At thehead of the queue is the transform for point M. Accordingly, by lines105 within block 106 is determined the vector chord MR2 as follows:

    .sup.T p.sub.chord =.sub.w p.sup.s (k)-.sub.w p.sup.s (k-n)

It appears that vector MR1 (denoted as D) is the projection of vector MOin the plane of the seam path. If MO and R1O are known, MR1, or D willbe known MO is known from the optical sensor system Center 0 is known,at 109 in FIG. 13D as _(t) ^(m) p^(cam). The relation between tool andsensor is known as _(t) p^(cam). The tool tip position is also known as_(w) T^(t). Therefore block 110 provides MO by the calculation: [_(w)T^(t) ]⁻¹ _(t) ^(m) p^(cam).

In order to determine what the vertical vector R10 is, first isdetermined a unitary vector u normal to the two vectors MR2 and o (k)(the orientation unitary vector at point M known from 104 for instant k,by line 105). This is done within block 107 using the result from block106 and the information retrieved by line 105 from queue 104 regardingthe o vector. Such orthogonal vector is obtained by the cross product

p chord x o (k)

In order to reduce it to unity (Un), this is divided by the scalar valuep chord x o (k). The result is outputted by block 107. R10 is calculatedwith u and with MO derived from block 108, by projecting the latter uponthe former through the operation: [_(t) p^(cam) ·U_(n) ]U_(n).

Since MO is the vectorial sum of MR1 and R10, D which is MR1 is obtainedfrom the vectorial difference _(t) p^(cam) -[_(t) p^(cam) ·U_(n) ]U_(n)as calculated within block 108.

Using the vector chord MR2 (from block 106 and line 113) and the vectorMR1 (from block 108 and line 112), block 114 and line 115 provide theangle cos Δψ therebetween according to the formula: ##EQU22##

Since the angle Δψ is a good approximation of sin Δψ, the value of line115 is converted within block 181 into Δψ=sin Δψ=I 1-cos² Δψ, with theresult derived on line 118. However, it remains to find out which signapplies, since the curvature with respect to vector o may be out to theright, or to the left. This is done within block 116 relating theorientation of vector to the orientation of vector D by: sgn [D. o (k)].Summer S1 in fact relates the angle Δψ of line 118 to the sign + or -.

As earlier explained with respect to FIG. 13C, the value of Δψ obtainedon line 131 is, within block 130, added to the values of line 129. Theyaw is not affecting the coordinate errors ΔPx, ΔPy or ΔPz. It affects,though, the orientation parameters of line 129, namely Δθ₁, Δφ₁, Δψ₁,for the pitch, roll and yaw by the respective amounts ax₁ Δψ, ay₁ Δψ,az₁ Δψ, as shown within blocks 120 and 130. The resulting errors whichare to be compensated by a corresponding control signal are Δx, Δy, Δz,Δθ, Δφ, and Δψ, appearing on line 133 and, after filtering through blockBLK3 of FIG. 13C, become the effective control signals passed via line136 through ALTER port 101 of the robot controller RB, or which arestored into queue 151 by line 150 for the three last ticks only.

The image processor invention will now be described by reference toFIGS. 23 to 38. As earlier explained by reference to FIG. 10, theoptical system provides in the plane of the array IA a 2-D image of thestripe containing point m of coordinates i and j. If kj is the verticalinterpixel distance and Jm the maximum vertical resolution of thesensor, and if ki is the horizontal interpixel distance, with Im themaximum horizontal resolution of the sensor, the light stripe image onthe sensor from the center point C on the image (i,j,) is Kj (j-Jm/2)and ki (i-Im/2) away in the two orthogonal directions. This image isproduced on a TV camera and it generates a standard RS-170 closedcircuit video signal.

A digital pipeline is provided to effectuate low level image processingleading to a light stripe image reduced to a one pixel wide linecharacterizing one edge of the stripe. A digital pipeline for imageprocessing is not new. See for instance "A Pipeline Fast FourierTransform" by Herbert L. Groginsky and George A. Works in IEEETransactions Trans.Comput vol C-19, pp. 1015-1019, Nov. 1970.

As explained hereinafter, the image processor according to the imageprocessor invention involves two processing levels. First, there is alow level image processing with the video signal from the camera,digitally through a pipeline, leading to a pixel line representation ofthe stripe line image as one edge thereof. Then, high level light striperepresentation is achieved with the processed pipeline output to lead tothe coordinates of the seam path embraced by the light stripe of theseam sensor.

Considering pipeline processing first, the camera operates in thefree-run mode and its sync signal drives the timing for the rest of theimage processing pipeline. As shown in FIG. 23, a pipeline, consistingof a series of solid state circuits on boards DG, VF, FS, SP, SNAP andFM, is provided to handle a digitized version of the video signal. Thelatter is received on line 201 from the camera CAM, while an analogreconverted signal is passed on line 202 to a TV monitor. The imageprocessor IP is centered about a microcomputer M68010 SBC, used tomonitor and control, over line 220 and a data bus (VME), all theoperations of the pipeline stages by bidirectional lines: 213 for boardDG, 214 for board VF, 215 for board FS, 216 for board SP, 217 for boardSNAP and 218 for board FM. All the boards communicate to each other fortiming, synchronization by line 212. In addition, the image processorcommunicates by line 220, data bus VME and line 221 with the controlprocessor CP of FIGS. 11 and 12. The control processor, typically, iscentered on another microcomputer M68010 connected to the robotprocessor via RS-232 ports (port 102 of FIG. 13A).

.Board DG phase locks by line 201 to the camera signal and generates allnecessary timing signals for the rest of the boards in the pipeline. Thesignal is digitized at, typically, a resolution of 512×512×8 at 30frames/second, each pixel being represented as a one byte quantity. Thedata stream is passed through one of 16 selectable and programmableinput lookup tables, the one used being made transparent by initializingit with a unit ramp function. The purpose is to suppress background andto sequentially translate the pixels of the picture from lines and rowsinto a chain of bits to be grouped, analyzed, transformed, or converteddigitally along the several stages of the pipeline, as explainedhereinafter. The connection with the next stage is via a ribbon cable203. A feedback loop is provided via line 204 to the digital input ofboard DG.

Next is a video 2-D linear filter VF. It carries out a finite impulseresponse (FIR) filter operation at TV frame rates with a 3×3 region ofsupport defining the size of a mask, or neighboring region, attached toone pixel. Only one filter board VF is shown, although it is clear thatseveral VF filter boards can be cascaded or filters with larger regionsof support can be used. The linear filter serves two purposes: reductionof camera noise variance and smoothing of the speckle caused by thelaser diode in the light stripe generator. The filter coefficients areselected to create a low pass filter, and an onboard barrel shifter isused to give the filter unity gain at D.C. The result is to enhance thedetectability of the light stripe by reducing the variance of the noiseand the variance of stripe plus noise without reducing the meanseparation.

By line 205 board VF connects with board FS and by line 209 to board SP.The object here is to eliminate streaks from spatter and sparks.Narrowband optical filtering does not eliminate the streaks caused bysparks. If spark trails occur only in one TV frame, camera lag wouldcarry over the streak signal for many frames, unless a lag-free camerais used, such as the CCD camera. It appears that the stripes havesignificant autocorrelation lag values, whereas the spark trails do not.Accordingly, boards FS and SP effectuate a filter function based ontemporal, as opposed to spatial, autocorrelation. To this effect, asshown in FIG. 24, board FS is a framestore board associated with a pixelprocessor SP. For the framestore board, in order to generateautocorrelation lags at TV frame rates (30 frames/second), two fullframe buffers BF1 and BF2 are provided, like shown in U.S. Pat. No.4,683,493. One holds data from the previous frame and reads it out tothe SP board, while the second frame buffer acquires the presentincoming video frame from line 205. The output of one frame buffer ismultiplied by the current frame, while the second frame is used to storethe current frame. Switch SW swaps the roles of the two buffers at theend of the frame. Therefore, the FS board always outputs a video frame,delayed, by one frame, to the SP board. The IP microprocessor by line215 supervises the swapping of the buffers. It also monitors themultiplexer settings and provides for compensation for pipeline delay.The video stream is a 8-bit per pixel format. One output from the VFboard goes directly by line 208 to the SP board, while the other by line205', or 205", enters the FS board. In addition to the one frame delay,there is a 10 pixel line delay in the data path via line 205 which doesnot exist via line 208. The two images must be correctly registered foreffective autocorrelation. Therefore, such 10 pixel time delay iscompensated by setting a 10 pixel advance for the buffer being read outto the SP board. The other buffer at this time has its pan register setto 0, in order that the interrupt task from the monitoring IP write twopan registers upon every buffer swap. Accordingly, two video streams areconcurrently entering the SP board, one from the VF board, the otherfrom the FS board. A one-frame-time autocorrelation lag is establishedon an instantaneous basis by multiplying the current image by theprevious one on a pixel-by-pixel basis. The SP board is a multiplieroperating upon pairs of pixels from the two 8-bit format video datastreams. The most significant byte of the 16-bit product is outputted online 210. It represents the instantaneous temporal autocorrelation lagfor the image. Besides eliminating spark trails, while not distortingthe light stripe image, this technique has the advantage of enhancingthe contrast in the image. This is due to the fact that the mean valueof the signal is larger than the mean value of the background noise. Inthat case, the ratio of the squares of the means will always be largerthan the ratio of the means themselves. Therefore, keeping only the mostsignificant byte after multiplication will eliminate a substantialamount of background noise if the noise means are low.

A smooth, low-noise image, which is virtually free from spark traileffects is outputted on line 210. As shown hereinafter, such image willbe segmented by amplitude thresholding (all values above 255 in pixelintensity being retained as a "white" and all values therebelow beingtaken as a "black") in order to produce a binary image resulting in afull scale white stripe line against a completely black background.Nevertheless, the stripe will be quite thick, due to the thickness ofthe original light stripe image and also due to the spatial blurringeffect of the 2-D low pass filter at the VF stage of the pipeline.

For further treatment, especially in the context of coordinatesextraction for seam tracking and robot control, it is desirable to havea stripe of narrow and precise width. To this effect, according to theimage processor invention, the stripe image derived on line 210 isreduced to the width of one pixel, exactly and throughout, namely at allpoints. This technique will be now explained as part of the digitalpipeline of FIGS. 23 and 24. FIG. 23 shows the succession of solid statecircuits, in the form of printed circuit boards, which form thepipeline. FIG. 24 is a block diagram providing a schematicrepresentation of the FS, the SP and the SNAP boards of FIG. 23.

Considering the SNAP board, the incoming digital video data of line 210is first passed through an Input Look-Up Table circuit IRAM, then, itgoes via line 220 to an input sequencing logic circuit PNG acting as apixel neighborhood generator, and the kernel so derived (a 3×3neighborhood in the example) is made available in a multiplex fashion,via ten parallel lines 225, to a comparator bank CMPB effecting theafore-mentioned thresholding function for the individual pixels andoutputting the resulting pixels on parallel lines 226 (all white above255 at the input, all black below 255). Lines 226 go to a look-up tablecircuit CRAM which, according to the image processor invention, retainsas a white pixel only those pixels which appear on lines 225 as one ofthe ten possible situations of FIG. 26A for which the lower edge of thestripe appears to border the two light and black zones. All otherneighborhood combinations (including those corresponding to the upperedge of the stripe) are translated as background i.e. as a black pixel.Typical of such other situations are the neighborhoods of masks 11, 12and 13 of FIG. 26B, respectively. The results appear on lines 227 whichare multiplexing the "pixel line" characterizing the lower edge of thelight stripe image of the camera which will be used for high levelprocessing, also according to the image processor invention.

The hardware circuit known as a SNAP, for systolic neighborhood areaprocessor, was published by Datacube, Inc., Peabody, Mass., 01960 (May,1986). This circuit is managed and controlled by the IP microprocessorso that, after thresholding by the CMPB circuit, the CRAM circuitprocesses the pixels of the ten lines 226 so as, according to the videosignal of line 210, detect and thin out the light stripe down to thedimension of a one pixel wide line representing the lower edge of thestripe image. In the SNAP, the IRAM circuit is composed of eightsoftware selectable banks of 256k-byte by 8-bit look-up tables. Theeight banks are typically loaded during initialization, and duringoperation the active bank is changed from one bank to another during ablanking interval without image perturbation. The input sequencing logiccircuit PNG creates a 3×3 neighborhood, namely eight surrounding pixelsabout a central pixel which is the present pixel passed by line 210 andcorresponding to one line and one row of the camera image being scannedfrom line 201 at the input of the pipeline. If the central pixel is C,the eight neighbors are shown in FIGS. 15 and 16 as NW, N, NE for theupper row, SW, S, SE for the lower row, and W, E for the closest pixelson the same line, according to the compass cardinal and secondarypoints. The pixels are connected in series by line 221 from left toright on a line and from one row to the next. The comparator bank CMPBincludes ten comparator units, also marked like the pixels of a cluster,the center being divided into two units CP (Center Primary) and CS(Center Secondary) which are both fed by the center pixel C. All theothers are fed by a corresponding one of the eight neighboring pixels.Lines 225 are the inputting lines to the ten comparator units of circuitCMPB. A reference threshold is established for each comparator so thatany pixel below the threshold will be outputted as black, whereas anypixel above the threshold is outputted as white. The output lines 226are multiplexed into the CRAM circuit receiving the single bit outputsfrom each of the ten CMPB comparators as the 10 address lines of theCRAM. The CRAM, typically, consists of two banks of 1024 by 8-bitlook-up tables, thereby translating the 10-bits of comparison outputdata into 8-bit results, and allowing 1024 possible input code addressesto provide 256 possible output codes. The two banks are loaded initiallyand during operation they are used interchangeably at appropriate times,i.e. upon vertical blanking. The operation of the SNAP is effected as a7.15 MHz process.

Considering the 3×3 kernel within the PGN block, along line 221 all thepoints are on the same line. The results of processing the center point(Primary and Secondary Center) are outputted nine pixels after thecenter point has been inputted. Delays are provided from all otherpoints in the kernel by adding or subtracting the appropriate number ofpixels counted toward or from the center point. Thus, the SNAP circuitintroduces a process delay of V+6 pixels to the center point of thekernel, V being the length of a line on the image. The results ofprocessing are outputted by the SNAP circuit V+6 after the center point(Center Primary and Center Secondary) has been inputted. The delayassociated with the other points in the kernel are obtained by adding(or subtracting) a corresponding number of pixels to (or from) suchdelay of V+6.

The kernel appears as situated within a sliding window. As the center Cmoves from one pixel to the next along the line to the left, point Ebecomes the new center C, while N, C, S become NW, W, SW respectively,and the contiguous pixels to the left of NE, E and SE become the new NE,E, SE. A similar shifting of the combination of nine pixels to form anew neighborhood occurs from one line to the next line below, causingNW, N, NE to move out of the picture, while SW, S, SE are belongreplaced by the three neighbors just below. The SNAP, while followingthe moves of the center pixel along lines column by column, takes intoaccount the whole picture of the kernel in terms of intensity of lightas translated digitally by the pipeline. It is understood that the sizeof the window is a matter of choice and practicality. Illustratively, a3×3 kernel has been chosen.

Thresholding is effectuated on each of the ten elementary comparators(NE to SW) of comparator CMPB, the reference input corresponding foreach to the level of intensity 255. All values of the pixel intensitythereabove will be translated on the corresponding line 226 as a white,all values therebelow will appear on line 226 as a black. The ten line226 are addressing the CRAM circuit.

Referring to FIG. 26A, ten situations identified as masks 1 to 13 arerepresented which represent all the possible combinations of black andwhite pixels when the window overlaps one edge of the stripe picture,here the edge below. FIG. 26B illustrates three among all the othersituations, identified as masks 11 to 13. According to the presentinvention, in each of those ten former instances, the center pixel SNAPoutput signal will be caused to be a ONE (for WHITE), whereas for allother instances, like in the later situations of masks 11 to 13, i.e.the other combinations of pixels in the kernel as seen through thewindow, the SNAP output code will be made a ZERO (meaning BLACK). FIG.27 is a table showing the encoding of the ten lines 226 as addressingthe look-up table CRAM for masks 1 to 10 of FIG. 26A and 11 to 13 forFIG. 26B. With this general rule of operation for circuit CRAM, not onlythe background of the stripe image which is not on an edge of the stripeas seen through the window will be black, but also the stripe imageitself to the extent that the window shows all white pixels or, with theupper edge (in the example), the pixel combination is the complement ofany of the situation shown in FIG. 16A. The stripe as digitallyoutputted by the SNAP will, then, appear like a one pixel wide line. Asgenerally known for a look-up table, the reference values whichcorrespond to the ten situations retained for the lower edge of thestripe are stored, and the inputted code from lines 226 will, as anaddress, recognize the location of the matching stored mask situation.With a 512×512×8 frames per second definition on the camera, there arein decimal equivalent 512×512 locations in the look-up table. The teninput lines 226 of FIG. 20 will form an address, as shown by FIG. 27,for the thirteen situations previously enumerated. For mask #1, forinstance, the ten lines binary code is the decimal equivalent 127, forwhich location the selection in the table prescribes a white at theoutput, namely 255 under the output code. In contrast, any of the lastthree situations lead to zero at the output. The overall result is online 211 a serially defined pixel line characterizing the lower edge(masks 1 to 10 of FIG. 26A). At this stage, the pipeline has completedthe low level processing of the image. Before considering the high levelimage processing which will lead to the derivation of the coordinatesfor queue 104 of FIG. 13C of the control processor CP, it should beobserved that, due to interlacing in the camera picture, thinning by theSNAP of the pipeline will totalize two pixels in width, rather than theexpected one pixel width for a single combination of lines and columns.Therefore, an additional step has to be taken in order to reduce theedge representation to one pixel as intended.

Referring to FIG. 25, the overall block diagram of the image processoris shown to include a low level treatment section LLT (comprising thepipeline PP1 and the 2 to 1 pixel reduction operator) and a high leveltreatment section HLT (comprising the additional steps). High levelprocessing consists in point sorting, stripe modeling via abstractrepresentation, shape matching, target refinement and finally 3-Dcoordinate conversion. These steps have been considered by the priorart. However, in this respect a new approach is now taken, according tothe present invention, due to the fact that, purposely, the stripe hasbeen reduced to a pixel line, which allows a fast and direct treatmentof the (i,j) coordinates of the points on the pixel line as received onthe 2-D camera picture

An important function not shown in FIG. 25 because, according to thepreferred embodiment of the invention, it has been included within the 2to 1 pixel reduction circuit PRD, consists in changing a 2-Drepresentation of the pixel line derived from pipeline PPL into a 1-Drepresentation thereof. This means that, from the 2-D grid of lines andcolumns (shown in FIG. 28A) which carries (as shown in FIG. 28B) thedistributed pixels of the image, for each pixel is derived, by column iand line j, a 1.D representation by discrete ordinates of value j foreach point a function j=f(i) of the abscissa i (as shown in FIG. 28C).This is simply done by placing within a table (RAM of FIG. 28D) the listof the values of j for each address i. The result is the afore-mentioneddiscrete function of i which as such can be treated with advantage, dueto speed and refinement, by the well known methods of numerical andalgebraical calculation, as will be shown hereinafter. This is done bythe microprocessor which is monitoring and controlling the operations ofall the blocks of the high level treatment section HLT of FIG. 25.

Considering the 2 to 1 pixel reduction, this is done within block PRDconcurrently with the 2-D to 1-D signal transformation. Due tointerlacing, the signal derived on line 211 at the output of the CRAMcircuit of FIG. 24, as shown illustratively on FIG. 28C represents whitepixels which are mostly double. With the RAM of FIG. 28B each pixel ofaddress i will be stored once with the first frame, then a second timewith the second frame. However, each time the second storing processencountered a white pixel for the same address, the preceding one iserased, i.e. the second storing replaces the first. The remaining pixelsare shown in FIG. 28C as shaded areas. As a result, it will appear thatnot only the "pixel line" has been reduced from a double to a singlepixel line but also, within the RAM of FIG. 28B, the 1-D discretefunction j=f(i), shown in FIG. 28D, has been established. Themicrocomputer CMPT calls, per column and per line, for each j value of awhite pixel of a column i, namely storing how high in the ordinateposition is the particular white pixel of abscissa i, and places suchvalue of j in the i location of the table listing within the RAM of FIG.28B. While so doing, each time that another j value is encounteredalready stored in the table for the same i location, the second value isstored in the table at that location and the previous one is deleted.The result will be that wherever a white pixel has been recorded for agiven line position, there will be only one pixel ordinate eitherrecorded only once, or stored the second time. It appears that withinblock PRD this process will have led to a reduction to one pixel lineand the 2-D representation will have been replaced by a function j=f(i)in discrete form. This is what is outputted on line 250 at the output ofthe low level treatment section LLT.

Except for the portion of the control processor CP where the queue 104receives on line 103 the samples defining the seam path locations assensed, the rest of FIG. 25 relates to the image processor, operatingunder control of the microprocessor CMPT. It is observed that queue 104is a RAM shared by both the control processor CP and the image processorIP. Nevertheless, arbitrarily, queue 104 has been shown, as in FIG. 13C,as pertaining to the control processor.

It is assumed that the seam tracker is proceeding with the laser beam toscanning across the seam, step by step along the seam path. Accordingly,from line 250, at the output of the digital pipeline, onto block GRF aseries of successive stripes reduced to a pixel line are passed. Eachstripe representation so derived is in the form of a discrete functionj=f(i). Block GRF is a function generator establishing under control ofthe microcomputer CMPT with data from line 250 a graph of the stripe,namely a 1-D graph which, according to the image processor invention andas explained hereinafter, is a symbolic representation of the stripe.

The operation of block GRF will now be explained by reference to thedetection steps performed successively by detectors ED, CRD, CRL and LADon the discrete function of line 250, upon each stripe step along theseam path being sensed. The purpose is to match the output of block GRFwith the outputted signal of line 302 from a standard graph derivedahead of time for comparison within comparator CMP3. FIG. 29 showsillustratively six possible light stripe images. Typically, stripes a/,b/ and d/ are characterized by straight lines. The seam corresponding tostripe a/ has an upper node and a lower node situated on the samevertical line, namely on the same column of the video image representedby the pixel line of line 211 of the pipeline. According to the imageprocessor invention, treatment of the image signal is done so that thesecond node will be automatically shifted forward by one column wheneverit is detected by the microcomputer that there are two successive pixelson the same column. Therefore, treatment and interpretation at the 1-Dlevel, beyond line 150, will be serial and unambiguous. The seamaccording to image b/ of FIG. 29 is a V-shaped seam. It is a U-shapedseam according to image d/. Images c/ and f/ are characterized by thepresence of arcs, rather than lines, between nodes, although image f/has both. Image e/ involves only lines but combines two seams each of adifferent type. The philosophy of the operation of block GRF will beunderstood by reference to FIG. 30 which shows graphs characterizing astripe image such a shown in FIG. 29. Corners, or nodes, are representedby a small circle, and the possibilities encountered are covered by theeight situations of FIG. 30. First, there may be an END followed by aLINE (under a/) or by an ARC (under g/). This situation will beautomatically detected at the beginning of the graph when followed by aLINE, or an ARC. The symbol is, thus, END/LINE, or END/ARC. Similarly,under b/ is shown the symbol LINE/LINE meaning an END encounteredbetween two straight lines, and under e/ the symbol ARC/ARC means an ENDbetween two arcs. FIG. 30 generalizes by showing eight situations whichare successively: END/LINE; LINE/LINE; LINE/ARC; ARC/LINE; ARC/ARC;LINE/END/; END/ARC; and ARC/END. An END is detected when ending a LINE,or an ARC, with another detection following for a LINE, or an ARC, orwhere there is no image before or after the detection of a LINE or anARC. Those symbols characterize the shape of the "corner", which itselfcharacterizes the shape of the seam to be worked on by the effector end.

The output image from the pipeline on line 211 consists of a largenumber of black pixels and 400-600 white pixels, representing thetwo-pixel thinned light stripe image. The positions of these whitepoints contain all of the surface reconstruction information. The objectof the high level treatment (HLT section of FIG. 25) is to extract thecoordinates of the white pixels from the image.

In previous systems, a general purpose processor would have been used toscan all of the pixels in the image, testing white and black, and savingthe coordinates of the white ones. This necessitates a frame buffer tostore the thinned line image, a very slow process, at 7 to 10 millionbytes per second, to scan the data and save the coordinates. Incontrast, the FM board (FIG. 2) scans the video data stream from theSNAP board and, on a per frame basis, extracts the coordinates of thewhite pixels. The extracted data appear in the form of a sequence of iand j coordinates, which are placed in a large table (RAM) accessible tothe MCPT (M68010 general purpose processor) which carries out theintelligent analysis of the image data. Each of the boards of thepipeline plugs into the P1 connector of a VME bus. A Mizar 7100 board isused including a single board computer (CMPT) which operates with a 10MHz clock (12.5 MHz), 512 bytes of random access memory, a real timeclock and two serial ports. The M7100 board supervises the pipeline lowlevel treatment (LLT) and the high level treatment (HLT) for theintelligent analysis of images. The M7100 processor is interfaced withthe control processor (CP) and shares with it RAM 104. The DG board actsas the video bus master, providing SYNC and timing signals to the restof the image processing boards. The FM board provides the terminationfor the video bus and acts as a slave on the video bus. All other imageprocessing boards are configured as non-terminating slaves on the videobus.

Considering now the high level treatment section (HLT) of FIG. 25, it isobserved that the surfaces which must be tracked by a practical roboticseam tracker are diverse and highly variable. Most prior art seamtracker designs have assumed explicitly, or at least implicitly, thatonly certain and few joint geometries are of importance. Furthermoreonly one geometry has been assumed to define the joint. This is contraryto reality. Too many shapes would have to be account for, and for agiven shape, lengths, angles and radii are definitely not constant.Therefore, another approach than those specific attributes is requiredfor a correct and consistent shape recognition procedure.

It is now proposed to base light stripe recognition on a feature-baseddescription of the stripe images. To this effect, the description isbased on capturing the invariant aspects of the light stripe images. Asearlier mentioned, lengths, angles and radii are not invariant featuresfor the description of a light stripe images. These are numericalattributes to be used in the final analysis, not to perform shaperecognition. Due to geometric constraints with a light stripe sensor, itis not possible for the stripe to have more than one sample detected inany given image column after final thinning. This is the purpose of thePRD circuit of FIG. 25. As a result, the three-dimensional imagetransformed into a 2-D image by the camera has been transformed into a1-D signal on line 250 which will be used to reconstruct the original3-D surface shape.

The line stripe signal is modeled in terms of the following features:(1) line segments (LINE); (2) circular arcs (ARC) and (3) corners,intersections of lines and/or arcs. Since arcs are difficult to extract,an option for the sake of simplicity is to assimilate an arc to a line.

Corners are the primary features to be extracted from the light stripes.They are classified into several types recognized while scanning fromleft to right in the image: (1) left end line; (2) line/line; (3)line/arc; (4) arc/line; (5) arc/arc; (6) line/right end; (7) end/arc;and arc/end. When two segments intersect to form a corner, the angle ofintersection is not invariant, due to natural variations in the surfacesto be tracked. The corner may be considered as "turning". However, doingthis "up" or "down" is not a rotationally invariant description of whathappens. In contrast, according to the present invention, the stripewill be described when it turns or bends at a corner, as doing a "leftturn" or a "right turn", regardless of the orientation at the corner.Accordingly, each corner is classified with this second attribute.

The preceding are symbolic attributes. In addition numerical attributesare also extracted from the raw light stripe of line 250. Thesenumerical attributes are used for two purposes. One is to facilitate theextraction of refined measurements which result in the generation ofsamples of surface position and orientation as well as gap size. Theother purpose is to facilitate the joint recognition process byeliminating certain corners, as will be seen hereinafter with theparsing operator POP of FIG. 25. These numerical attributes are: (1)corner horizontal coordinate; (2) corner vertical coordinate; and (3)corner normal distance.

As will be shown hereinafter, the ultimate goal with the image processorinvention is to determine by detecting LINE, ARC, or END, or otherdefined geometric characteristic, and combination thereof, whatcharacteristic there is, how many of them there are, for instance howmany corners between lines or arcs. From such data the shape can beidentified. Where the purpose is to locate a point of reference on ajoint or seam path, the important feature is the corner. From theidentified shape, it is known where the preferred "corner" is, that is,the "target" for the effector end to work on and, therefore, where forone particular stripe image the seam location as sensed actually is. Inother words, in the final analysis the coordinates of the target aredetermined which will be used in the context of the control processorinvention.

Referring again to FIG. 25, the signal of line 250 (see FIG. 24D) whenreceived on line 251 provides at the start, by END detection withinblock ED, an indication thereof passed by line 251' onto block GRF.Thereafter, corner detection block CRD detects from line 252 the cornersexisting throughout the signal of line 250. To this effect, the functionof block CRD is carried out by a discrete signal difference operator.Referring to FIG. 28E, this operation, which is conventional innumerical calculation, consists in ascertaining the difference betweensuccessive values of the discrete function j=f(i) and in detecting when,between two successive such differences, the value jumps by apredetermined minimum amount c. In order to eliminate noise due toerratic distribution, instead of operating upon the values immediatelypreceding and immediately succeeding the maximum (in the illustration,since in another illustration it could be a minimum), i.e. the change ofslope, values at a rank s on both sides, typically, are used. Therefore,the detected difference results from the computation:

    c=f(i)-f(i-s)-[f(i+s)-f(i)]=2f(i)-f(i-s)-f(i+s)

Having identified, upon such successive detections of a maximum, or aminimum, of the function f(i), how many corners there are as they aresuccessively encountered, the next steps consist With operators such asCRL in FIG. 25 (lines 254, 254' and 256, 256') in establishing therelation of each such corner with the preceding, or succeeding, slope ofthe line segment connected to it. This is based on the same functionj=f(i). To this effect, it is established (thus, with a firstderivative) whether, from such corner, the pixel line is turning RIGHT(negative slope), or LEFT (positive slope). Moreover, based on twosuccessive corners, it is determined whether these are connected by aLINE, or by an ARC. This is done within operator LAD (lines 253, 253'and 255, 255'). Again, simple algebra in treating the discrete points ofcurve j=f(i) will reveal a linear succession for LINE, or a curvaturefor ARC. The last step is, at the end of the pixel line, to detect anEND (lines 257 and 257').

Segmenting point files into lines, arcs, . . . is quite slow. Forinstance, the Hough transform approach requires the generation of alarge number of points from edge point in the point file, according to alinear, or circular, equation. Recursive line splitting is anotherapproach. The latter takes about 100 milliseconds to run, for typicalshapes and lines only. In contrast the approach according to the presentinvention takes only 8 milliseconds to run.

Referring to FIGS. 31A and 31B, a graph header GRH is provided formonitoring the buildup of the symbolic graph, typically, constituted byfive RAM's associated with the respective "corners" (extreme ends to theleft and to the right, and E, F and G, in the case of FIG. 31A, onlyfour RAM's in the case of FIG. 32A). The RAM's are tail and endconnected, and each has corner identifying features relative to adjacentsegments of the stripe line. Thus, RAM1 contains the symbol (L-END forend/line, as under (a) in FIG. 30), the type E-L (for END/LINE, relativeto the segment until point E in FIG. 31A which is the stripe image), thevalue of c (marking the acuteness of the change of slope), thecoordinates of the corner, the turn on passing over the corner.

Having identified within block GRF a pixel line having so many cornersand noted how these are connected, it remains to sort out the truecorners from those which, as shown in FIG. 34A and 34B may be doublecorners. There, is shown with FIG. 34A a pixel line comprising twocorners F and G placed on the same vertical line between an upper lineEF and a lower line GK (as explained earlier, actually G is placed onecolumn further than F in the process of deriving the pixel line). Asshown by FIG. 34B because of double corners (F, F' and G, G'), itbecomes necessary to ignore one in each pair and to use only one,thereby being in the same situation as in the true seam of the curve ofFIG. 34A. This is the function of the parsing operator POP of FIG. 25.Referring to FIG. 31A or to FIG. 32A, it is seen that between threecorners E, F, G, two corners are aligned on a line such as EG for E andG, and the third corner, F in this instance, is on the normal FH to theopposite side of the triangle. Knowing E and G, line EG is calculated,and the distance FH to line EG is also calculated. If we compare FF', orGG', of FIG. 34B, it appears that the same triangle approach will revealdouble corners. Therefore, the parsing operator POP will cancel theindication of one of the two close-by corners, thereby restituting thegeneral shape of the line as in FIG. 34A.

How the operation of graph block GRF and operator POP of FIG. 25presents itself will now be considered by reference to FIGS. 31A, 31Band 32A, 32B. The pixel line as received on line 250, in the form of thefunction j=f(i), is read by the microcomputer from left to right interms of the graph of FIG. 31A, or FIG. 32A. Considering for instanceFIGS. 31A and 31B, the graph header GRH, which looks successively foreach light stripe occurring while the sensor is following the seam (seeblock SLC of FIG. 11), opens a file for the particular pixel line (FIG.31A) under investigation. The object is to recognize the key features(shown in FIG. 31A) and to store the relevant information in such file.The first operation is corner detection (block CRD of FIG. 25),accomplished by detecting i.e. the rate of change in slope detected bythe second difference operator already considered by reference to FIG.28E. It tells how many corners there are. Accordingly files RAM1, RAM2,RAM3, RAM4 and RAM5 are opened, one for each corner. Then, these filesare considered one after the other for more characteristics.

First, there is the first END to the left. In file RAM1 besides c=0 arenoted the coordinates of the END, namely i=10 and j=250. Besides thosevalues are stored, as shown in RAM1 of FIG. 31B, the symbol "left end"(L-END) and the type (E-L like under (a) in FIG. 30). Since theoperation is progressing to the right from the start, there is no linkto the rear, the system looking for another corner toward the right.Therefore, there is also a "link forward" from this first file to thebeginning of the next file which is for the subsequent corner.

File RAM2 was opened during the corner detection operation (CRD of FIG.25) upon the detection of a change in slope characterizing corner E. Atthe same time, it is noted in the file that for corner E: i=50 andj=250. Now, block CRL (FIG. 25) determines for corner E whether there isa turn to the right, or to the left. Slope detection indicates that theturn is to the right. The file is given this information. The same isdone for corner F (RAM3), and for corner G (RAM4). It is possible todetermine whether up to corner E, between E and F, between F and G, andfrom G to the extreme end to the right (RAM5) whether those corners areinterconnected by lines (FIG. 31A) or by arcs (FIG. 31B). This is thefunction of blocks LAD of FIG. 25. A simple approach is to assume thatthere is a line in between. The symbols (FIG. 31B) are L-END (left end)for RAM1; RT-TURN (right turn) for RAM2; LF-TURN (left turn) for RAM3;RT-TURN (right turn) for RAM4 and R-END (right end) for RAM5. Forwardlink (f link) and rear link (r link) are also noted in the files,thereby to indicate the line of continuity between the successive filesas one would read the graph of FIG. 31A, from left to right, or fromright to left. The graph pointer (graph ptr) in the graph header GRHindicates which file to go as the sensor moves. The track pointer (trackptr), also in the graph header, indicates where the critical corner, ortarget, for the effector end is.

Before matching the graph so identified with a standard graph (a seammodel of block SMD passed for testing to block SGF for matching bycomparator CMP3 of FIG. 25), there is a need for a parsing processperformed by the parsing operator of block POP in FIG. 25. This is dueto the fact that the corner detection process generates more cornersthan is desirable. Typically, double corners (such as shown at F' and G'in FIG. 34 under (b)) are generated by the first corner detectorwhenever step continuities appear in the stripe image. The effect of thecorner parsing process is to scan the corner structure array and deletethe extraneous corners. The deleting process, referred to as parsing,namely to reduce the original corner string to a simple one formatching; may be performed according to a variety of parsing rules.According to the preferred embodiment, in the parsing process, a newcorner attribute is computed which is the normal distance from thecorner to the straight line between its two adjacent corners. Thedistance (such as FH in FIG. 31A) from the corner (F in this case) tothe line joining the adjacent corners (EG in FIG. 31A) is established.Should there be a double corner, like in FIG. 34 the second corner is F'assumed to be very close to corner F, the same process will provide thedistance FH' to line EF' (the two adjacent corners being now E and F',rather than E and G). As a result the distance FH' will be very small.The rule is to have a minimum distance below which the middle cornerwill be ignored for all future use. Therefore, corner F in this casewill be cancelled. This leaves the next and close corner F'. Now, thedistance is calculated from F' to E and G, since F has been cancelled.The process can be continued after such elimination of the doublecorners F and F', under the assumption made for illustration. Moregenerally, the rule is to determine upon each corner whether there is avery short distance (in a case like FF', or GG,' in FIG. 30 this willappear from the very first corner of the pair) and to eliminate suchcorner immediately, thereby making the next corner of the pair a singlecorner for subsequent computation. FIG. 33 shows for the five corners 1to 5 (RAM1 to RAM5 of FIG. 31B) how the symbols are progressivelyestablished on the basis of the corners starting from the extreme onesand narrowing with the intermediary ones from the left to the right ofthe overall graph. The normal distance is computed according to thefollowing equation: ##EQU23## where a=(y2-y1) and b=(x2-x1) with (x1,y1)and (x2,y2) being the coordinates of the corners adjacent to the centralcorner (x,y). The successive h values 0, 23, 200, 18 and 0 are stored inthe respective RAM's.

Once the value of h has been computed and the conclusion andsimplification done, the corner symbols may be assigned (RAM1, RAM2,RAM3, RAM4 and RAM5 in FIG. 31B). It is observed here that the sign ofthe distance h can be used, instead of the slope, in order to determinewhich direction is taken (going from left to right in the process ofestablishing the graph). Therefore, in such case, the turn direction(RT-TURN or LF-TURN) will be defined by the sign of h.

It appears that, once filled with the data so obtained for theparticular light stripe, the RAM's (FIGS. 31B or 32B) will containnumerical attributes such as i, j and h, and symbolic attributes such asL-END, E-L LF-TURN, RT-TURN, f link, r link. The result on line 260(FIG. 25) at the completion of the parsing routine is a simplified graphof the light stripe image complete with values for all numerical andsymbolic attributes. The symbolic attributes will be used by thecomparator CMP3 of FIG. 25 to establish matching with the symbolicattributes of the standard graph passed as reference on line 302 undertesting.

In graph matching, joint geometries are recognized by matching referencegraphs (SMD in FIG. 25) to the graph extracted (line 260) from eachimage (SLC in FIG. 11) in real time. Such process is complicated by thefact that one joint may have several representations due to changes inmorphology along the path (what is quite common with metal sheets) anddue to the fact that start, stop and branch points may have drasticallydifferent morphologies. The solution to this problem is to allow eachjoint to be described by an arbitrary number of corner graphs. Eachcorner graph contains the string of attributed corner structures indouble linked list form. The set of all the corner graphs for a givenjoint is also linked together to form a tree. In the matching process,all corner graphs in a tree are compared to the sample until a match isfound, or all the references have been used.

Single graph matching is performed by stepping through the corners inthe reference graph (line 302) and the sample graph (line 260) together.In order for a graph to match a reference, the following conditions mustbe met: (1) the number of corners in the two graphs is the same; eachcorner in the sample has the same symbol (turn direction) as thecorresponding corner in the reference; each corner is of the same type(line/line, arc/line, . . . ). While the tests for matching are beingperformed sequentially, the sample tracking feature pointer of thegraphic header (GRH in FIG. 31B) is set when the tracking feature corner(target) in the reference graph is encountered.

Tree matching is carried out by attempting to match on individual cornergraphs of the tree in sequence until one matches or all choices havebeen exhausted. It is observed that, in practice, average matching timeis under one millisecond.

Once there is matching, comparator CMP3 will gate GTE to pass the graphfrom line 261 onto line 262, and from there on the numerical attributeswill be used. Knowing what type of graph corresponds to the seam undersensing, in accordance with the standard graph, by line 267, it isproceeded at TRS with the determination of the target, namely, theselection of the corner which is of interest both in working with thetool on the joint and in knowing where the seam path location is located(which means between the values of (i,j) in the RAM's which one willserve as the target coordinates, i.e corner F in FIG. 31A).

Eight symbolic attributes are represented in FIG. 30, as can be storedin the RAM's (FIGS. 31B or 32B). To these are added the "left turn" or"right turn" attributes. The numerical attributes are (i,j), the cornerhorizontal and vertical coordinates, and h the corner normal distance.

Besides the RAM's, in the process information is also stored in theheader GRH (FIG. 31B). It relates to three additional attributes bearingon the overall graph, rather than the corners. These are determinedafter matching.

One attribute is the tracking feature, specifying one of the corners inthe graph as the corner which represents the target. The coordinates ofthe target are used to generate sequences of homogeneous coordinatetransformations representing the seam path in 3-D, space coordinates.These transforms are used to generate control functions for the robot.As earlier mentioned, the header (GRH) for the corner graph (RAM1 toRAM5 in FIG. 31B) contains a pointer which indicates which corner is thetracking feature. In the reference graphs, this pointer is initializedby training. In the sample graphs, it is null until matching has beensuccessful.

A second attribute of the whole graph is the target refinement codespecifying which of several heuristic method should be applied whenusing raster data to refine the target corner coordinates. The code isinitialized by training for each reference graph.

A third attribute is the action code specifying what the controlprocessor CP should do whenever there is a match for the particulargraph of line 260. Typically, these actions are: MOMATCH (errorcondition, do not generate a control vector from the current sample);MATCH (generate a tracking control sample for queue 104); START (executethe start-joint sequence); STOP (execute the end-of-joint sequence);BRANCH (execute the branch to new path sequence). Each joint may have adifferent control action. The values are initialized by training.

These three attributes are added to the graph header GRH (FIG. 31B).

Once a target corner has been selected by graph matching, the originalraster data is re-analyzed in order to refine the coordinates of thetarget. This is the function of block CRF of FIG. 25 in response to line263. This done by reference to data from the reference graph derived online 268. The basic method consists in using linear regression to fitequations to raster data. This has a spatial averaging effect in theminimum mean squared error sense. The linear equation, or equations,which are generally known, are used to get refined target coordinatesaccording to one of several rules: (1) intersection of two adjacentlines; (2) left hand or right hand "carpenter's square solution"; (3)best left hand or best right hand fit; and (4) original corner. FIG. 35illustrates these four situations with lines such as L1 or L2, a normalline N, and a vertical line H, having an intersection on the targetpoint C. Typically, the carpenter's square solution deals with rounded,worn or scaled corners on the beam edges, as shown in FIG. 35 for theparticular example.

At this stage, the information derived on lines 264 and 264' of FIG. 25is passed to block GDM where, based on information received on line 269from the sample graph, the gap is measured. This is a distance fed tothe robot (by line 265 and via the control processor CP) toautomatically adjust the weld process to fill the gap with metal. Thisis known as "adaptive fill", although in the present description controlof the robot is by feedforward, and not adaptive in a theoretical sense.

After the target point coordinates have been refined, according toanother aspect of the image processor invention, they are directlyconverted to 3-D coordinates in the camera coordinate system (for lateruse by the control processor CP with the queue 104). The relevantequations have been mentioned earlier by reference to FIG. 10. In orderto convert data points rapidly, advantage is taken of the decoupling ofthe horizontal and vertical coordinates and of the limited dynamic rangeof the image coordinates to generate lookup tables for the functions.The standard method would consist in going through the followingtransformations: ##EQU24## which require twelve multiplications andeight additions, in order to carry out the matrix multiplication. Thenew method makes z and y (FIG. 10) functions of the image ordinate Jmand x a function of z and the image abscissa Im. As earlier stated andas shown in FIG. 10, the functions are as follows: ##EQU25##

This is based on x being the unit vector in the space x-direction asshown in FIG. 10, and n a unit vector normal to the light stripe plane.Therefore, the constraint is (x . n)-0, where the sign (.) representsthe vector dot product. Then, y and z are functions of the imageordinate Jm with constants, whereas x is given by z times a function ofthe image abscissa Im, also with constants.

This method can be implemented in dedicated hardware, in embeddedfirmware, or in general purpose software. It involves three table lookuptables, xLUT, yLUT and zLUT (FIG. 36), and one multiplication, at S1 inFIG. 36. The address j is passed by line 400 to the yLUT lookup tableand by line 401 to the zLUT lookup table, the output coordinates y and zbeing on lines 402 and 403, respectively. The i address is passed to thexLUT lookup table and its output on line 405 is multiplied by z asderived from lines 403 and 406. Coordinate x appears on line 407. The xcalculation is slightly more complex, since integer arithmetic is usedfor speed of operation. Therefore, the xLUT lookup table requires ascale factor in order to expand the dynamic range of the normalized(z=1) x function. This scale factor must, then, be divided out atconversion time, although it can be a power of two for efficientimplementation. As to the m_(zx) component of surface orientation, itmay be somewhat more complex. As shown in FIG. 37, three lookup tablesgLUT, g2LUT and hLUT are used, with two additions at S6 and S8, withfour multiplications at S3, S4, S5, S7, and with one division at 430. S8involves the product of i, derived from line 410 by Im/2 derived fromline 411. The output goes by line 412 to multiplier S3. The three lookuptables receive the address j from line 413 and respective lines 414,415, 416. They output on respective multipliers S3, S5, S7 by lines 420,426, and 423, respectively. Line 420 like line 412, goes to multiplierS3 and the output is on line 421. Multiplier S4 responds to line 421, S5to line 426 and S7 to line 423. In addition, Δj by lines 417 and 418brings the second input to S4 while by lines 417 and 419 it goes as thesecond input to S5. Similarly, Δi by line 424 becomes the second inputto multiplier S7. Summer S6 combines line 422 from S4 and line 425 fromS7 to form the divisor B, whereas line 427 from S7 inputs the dividendfor divider 430. The latter outputs the result m_(zx) on line 429. Thisapproach allows a 46% improvement in throughput after redundancies inthe equation have been considered. A plain calculation would require28.5 equivalent additions, whereas the lookup table method requires only19.5 equivalent additions.

As a result, on line 266 are derived the position and orientation ofpoints on the seam: x, y, z, θ, φ, ψ. Considering block SCC, thehomogeneous coordinate transform is a matrix composed of four vectors,one indicating the position, the remaining three being orthogonal unitvectors indicating the orientation. As earlier stated, the hctm isdenoted by the symbol T with sub and superscripts to indicate relativecoordinate frames Thus, the transformation from robot world coordinatesto a seam sample is denoted: _(w) T^(s), the variations in the presentuse being s for seam; w for world; t for tool; and c for camera.

The image processor measures the positional offset component of seamposition directly in camera. coordinates. These values go directly intothe camera-to seam transform. The image processor measures m_(zx)(derived on line 528 in FIG. 38, which is the surface slope componentdefined by ##EQU26## which is related to the image slope Δj/Δi of line523 in FIG. 38, when the strip projector is aligned to place linesparallel to the i-axis when a flat surface is normal to the lensprincipal axis as seen in FIGS. 9 and 10) directly from a single frame.After collecting several samples of seam sample position while inmotion, it also computes mxy and mzy directly. These values are used tocompute the normal (n) and orientation (o) vectors of the orientationsubmatrix of the hctm as follows: ##EQU27##

The approach (a) vector is computed as the vector cross product of n ando.

As an illustration, in Appendix is given a fortran program for lightstripe design relative to light stripe geometrical analysis.

Referring to FIG. 38, a flow chart illustrates the overall operation ofthe image processor of FIG. 25. Block 500 typifies the RAM of FIG. 28Bwhich lists the values of j for every column i in the data raster of thepixel line outputted by the pipeline and, in the case of interlacing,block 502 responds to line 501 to provide the one-pixel line of line250. Thereafter, by line 503 the system goes to 504 where, as in CRD ofFIG. 25, corner detection is performed. Preliminary graph generation, aswith blocks ED, LAD, and CRL of FIG. 25 leading to the RAM informationcontained in block GRF, is performed at 506. The rough graph, outputtedon line 258 which in FIG. 25 goes to the parsing operator, is worked outby pruning in the graph parser at 508. From there, by line 509, thesimplified graph is operated on by the final attribute calculator at510. From there, by line 511 the final attributed graph goes to matchingat 513 (in response to the tree of reference graphs derived from line512). It is understood that the reference graphs derived for testingwith the actual graph of the light stripe image as sensed have the samegeneral attributes of a possible actual graph derivation. They have beenprepared in advance on the basis of seam models chosen as standards.After matching at 513, gap extraction follows by line 514 at 515, andgap code scaling occurs at 517 to yield the gap code on line 265 for thecontrol processor CP. On the other hand, the graph matcher of 513 leadsto target refinement performed at 520 (like at CRF in FIG. 21) providingthe target coordinates i on line 521 and j on line 522. The image slopeis also derived on line 523.

Then, at 524 (like at CRV in FIG. 25) conversion from 1-D to 3-D iseffected, thereby providing the three coordinates x, y, z on lines 525,526 and 527, respectively. The image slope is also converted intosurface slope m_(xz), as inclined on the y axis. The three coordinatesx, y, z are obtained as earlier stated from the following: ##EQU28##while the surface slope m_(zx) is carried out with total differentialsdz and dx, as generally known. The surface slope is derived on line 528.

The information so derived is thereafter assembled into hctm's, in orderto provide seam samples. The basic matrix for the seam transform is:##EQU29## where o is the orientation vector, a the approach vector and nthe normal vector, p being the position vector between camera and seam,as earlier stated with reference to FIG. 18. Using the data of lines 525to 528, the matrix is: ##EQU30## This is the output of line 530 fromblock 529. It remains now to convert from seam coordinate into worldcoordinate before instructing the control processor to control the robotprocessor. The relation is:

    .sub.w T.sup.s =.sub.w T.sup.t :.sub.t T.sup.cam :.sub.cam T.sup.s

wherein the transform _(W) T^(t), namely the tool coordinates, is givenon line 137 from the robot processor via the control processor. As totransform _(cam) T^(s) it is inputted by line 530 into block 532 and_(t) T^(cam) is a given from the sensor to tool fixed relationship. Theresult on line 533 is _(w) T².

Since:

    .sub.w T.sup.s =.sub.w T.sup.t :.sub.t T.sup.s

this is effected, with line 533 and with line 534 (derived from line 137of the control processor), within block 534, thereby providing _(w)T^(s) (k) for instant k on line 535.

The seam path coordinates are outputted on line 103' for the queue 104.The elapsed distance relationship thereto, is established for all thesamples at 536 and the sampled locations by line 103 are stored in thequeue 104.

It is to be understood that several original features have beendescribed in the context of the image processor invention asconstituting the preferred embodiment of the invention. Alternativesolutions are not excluded which concur to the same goals and unexpectedresults.

To summarize, the original features of the image processor invention maybe listed as follows:

A sensor of geometric form taken into account in ascertaining the seamsample coordinates;

An assembly of interconnected electronic boards for processing lightstripe images and converting the video signal into one-pixel wideimages;

A method of generically representing light stripe image structure interms of abstract data structures;

A method of matching data structures against reference structures sothat a match can be found in spite of distortions in the numericalattributes such as lengths and angles;

The ability to match against any of several reference models so thatessential morphology changes can be handled dynamically andasynchronously;

The ability to refine data interpretation in the context of a match soas to extract refined numerical attributes such as target position andgap magnitude;

As to the control processor invention, the following original featuresmay be summarized as follows:

The ability to have a robot move the sensor anywhere in the work volumeof the robot, searching for the start of a seam, and to halt the robotonce the start point has been found;

The ability of the teacher to cause the robot to "center up" theeffector end with respect to the seam start point in all six degrees offreedom before initiating tracking and taking into account desiredeffector end reference offsets;

Feedforward-based six degree of freedom tracking in real time withdynamically update of the robot path, dynamic recovery, andextrapolation of the robot path by vision system;

And, indeed, the integration into a single robotic system of the controlprocessor invention and the image processor invention describedhereabove is a major feature of the robotic system here proposed.

It is to be understood, that the invention has been described in thecontext of the preferred embodiments. This does not exclude, however,other embodiments using the essential features of the present inventionunder one, or other aspects thereof. For instance, the recovery of thetaught path from an industrial robot has been described as an importantfeature. Nevertheless, feedforward control according to the controlprocessor invention may also be performed with a robot wherein thetaught path is available. Also, the seam tracker has been describedusing an optical sensor. It is also possible to use other forms ofsensing system providing data regarding a discrete series of locationson the seam sensed ahead of the tool, for use by the control processoraccording to the present invention. It is also not necessary that thesensor be mounted on the tool at a definite distance therefrom.Correlation by elapsed distances between the sensed locations and thetool position and orientation can be established otherwise taking intoaccount the particular relationship therebetween. ##SPC1##

I claim:
 1. In a robot system including a robot controller forcontrolling an effector end along a seam in relation to a taught pathand seam tracker control system, the seam tracker control systemincluding a sensor system, for sensing the seam ahead of the effectiveend and for deriving data representing discrete successive sensedlocations upon the seam, and a control processor responsive to saidsensor system and to said robot controller for deriving a control signalrepresentative of an error between effector end positioning and amatching sensed location, the effector end being controlled by saidrobot controller in response to said control signal to provide seamtracking with the effector end;the combination of: means responsive tosaid robot controller and associated with said control processor forsuccessively calculating the distances between two consecutive sensedlocations, for accumulating said distances to provide iteratively theelapsed distance down to the last sensed location and for providingorderly samples each representing such elapsed distance to identify therespective sensed locations upon the seam; look-up table means forstoring data representing coordinates derived by the sensor system foreach of said sensed locations in relation to and in the order of saidorderly samples; the control processor including means for grouping saidlook-up table data in accordance with a common algebraic characteristicdefining a seam model; means within said control processor successivelyresponsive to a present and a last position of said effector end forextrapolating therefrom to an anticipated position and for deriving ananticipated elapsed distance thereto; means within said control processoperating with said grouped look-up table data for correlating saidanticipated elapsed distance of the effector end with a correspondingsample of said storing means to derive corresponding sample of saidstoring means to derive coordinates of an anticipated effector endlocation on the seam; comparator means within said control processor forderiving a present control signal as said control signal and in relationto said corresponding coordinates; and the control processor controllingthe robot controller with said present control signal so as toeffectuate feed forward control of the effector end toward a newposition defined by said anticipated elapsed distance; wherein the erroreffector end positioning and matching sensed seam location is appliedcyclically as a first signal by the control processor to the robotcontroller and is compensated by the robot controller in responsethereto through concurrent control of the effector end in relation to ataught path therein; wherein the robot controller cyclically, andfollowing said effector end cyclical control, provides the controlprocessor with a first signal representative of such present position;wherein the control processor cyclically generates with said first andcontrol signals successive second signals representative of successivecorresponding present positions characteristic of a recovered taughtpath; wherein said extrapolating means provide with consecutive saidsecond signals for a present and a last position characteristic on saidrecovered taught path, a third signal representative of an extrapolatedposition on said recovered taught path; said seam model being responsiveto said third signal for deriving a fourth signal representative of acorresponding seam location; said comparating means being responsive tosaid third and fourth signals for deriving said present control signal.2. The system of claim 10 with said sensed seam locations being groupedin response to a predetermined minimum change of slope betweenconsecutive derived sensed seam locations.
 3. The system of claim 2 withsaid common incremental characteristic being derived by the controlprocessor from successive sensed seam locations in accordance with theleast-squared error line method.
 4. The system of claim 2 withsuccessive predetermined minimum changes of slope are detected as anindication of a circular seam path.
 5. The system of claim 2 with saidcorresponding coordinates including a vector o representing theorientation of the effector end for the present position, theorientation vector o being successively determined by the controlprocessor as a function of the elapsed distance and in relation to thesame model grouped look-up table data, with the effector end anticipatedposition and orientation for control thereof being derived in accordancewith said model common algebraic characteristic and for said anticipatedelapsed distance.
 6. A method of operating a robot system including:arobot controller operating with a taught path; a control processor foriteratively controlling an effector end with an error representingcontrol signal in relation to a recovered path derived by the controlprocessor from said robot controller; a seam sensor vision having alook-ahead optical sensor for deriving successive 2-D light stripeimages representing discrete successive transversed views of the seam assensed at successive locations upon the seam a predetermined distanceahead of the effector end; an image processor for treating each of saidstrip images in real time to derive in relation to each image a firstsignal representative of the coordinates for the respective sensedlocations; and a control processor responsive to a second signalindicative of the present effector end position and to said first signalfor generating said control signal as an indication of an error betweenpresent effector end position and corresponding sensed location; withthe following steps being performed by said control processor; storingand classifying said first signal coordinates for sensed locationsobtained ahead of the effector end as a function of the elapsed distancesince a first location sensed at the start of the seam sensing process;said first signal coordinates following each other in the succession ofsensed locations to form a discrete series of signals of differentmagnitudes; grouping such discrete signals according to a commonalgebraic characteristic typical of a particular seam model; storingsuch seam model grouped discrete signals as a function of thesuccessively encountered sensed locations; said control processorperforming the following additional steps: calculating and storingiteratively in response to said control signal and to said second signalthe elapsed distance of the effector end since a start position thereof;deriving iteratively with said second signal for a present effector endposition and with said control signal for a present positioningoperation of the effector end successive positions representing thetaught path as recovered therefrom; extrapolating by a predeterminedamount from past positions on said recovered taught path to ananticipated position thereon; extrapolating by said predetermined amountfrom past elapsed distances to an anticipated elapsed distances for saideffector end; determining with corresponding seam model grouped discretesignals in relation to said anticipated elapsed distance the coordinatesof the corresponding sensed locations and deriving said first signalaccordingly; and controlling by feedforward the robot controller inresponse to the control signal derived from said first and secondsignals.
 7. The method of claim 6, with said seam model grouped discretesignals defining within said seam model at least one of the slope andthe change of slope as the common algebraic characteristic thereof.
 8. Arobotic control system including:a robot controller operative toposition an effector end in relation to a seam; an image processoroperative with an optical seam tracker and a look-ahead sensor mountedon said effector end a distance ahead therefrom to provide successivesensed locations on the seam; and a control processor responsive to aninstantaneous present position of the effector end and operative uponsaid robot processor for control of the effector end toward ananticipated position by feedforward in relation to a corresponding seamlocation derived from said image processor; the image processorproviding an indication of the successive sensed locations on the seamcounted at respective elapsed distances from a starting point on theseam; an indication of the successive effector end positions beingderived from the robot processor and received by the control processorand counted at respective elapsed distances from said starting point;wherein said corresponding seam location is ascertained by the controlprocessor in relation to successive sensed locations derived from theimage processor and to an elapsed distance corresponding to saidanticipated position; wherein the robot has a taught path and thecontrol processor derives from the robot processor a recovered taughtpath, an extrapolated elapsed distance being ascertained on saidrecovered taught path by the control processor matching said anticipatedposition, and wherein the corresponding seam location is derived fromsaid sensed locations as a function of said extrapolated elapseddistance; said sensed locations being stored and grouped within thecontrol processor in a function generator so as to form as a discretefunction of elapsed distances a seam model; said extrapolated elapseddistance being applied by the control processor to said functionsgenerated as a variable representative signal so as to derive a signalrepresentative of said corresponding location.
 9. The system of claim 8,with said function generator containing a plurality of said seam modelsformed successively by the control processor with the sensed locationsderived by the seam sensor ahead of the effector end, a selection of aneffective one of said seam models being effected by the controlprocessor in relation to said extrapolated elapsed distance.
 10. Thesystem of claim 9, with a seam model being characterized by a linearfunction of said discrete elapsed distances representing the slope ofthe seam.
 11. The system of claim 10 with said seam models definingsuccessive changes of said seam slope from one another with theoccurrence of said discrete distances in succession.
 12. The system ofclaim 8, with feedforward control of the effector end being performedfrom a present position related to the recovered path by reference tothe associated corresponding seam location, said extrapolated effectorend position being determined from said present position and saidanticipated elapsed distance being derived in relation thereto, saidcorresponding seam location defining an error used in the feedforwardcontrol in relation to said recovered taught path.
 13. The system ofclaim 12, with such feedforward control being effected cyclically, andthe elapsed distances being determined and stored by the controlprocessor upon each new effector end position and extrapolated position.14. The system of claim 13, with the look-ahead sensor being ahead ofthe effector end by a predetermined distance, with the succession ofelapsed distances for the effector end successive positions and thesuccession of elapsed distances for the sensed seam locations beingrespectively displaced from one another by the amount of saidpredetermined distance.
 15. A method of combining the operations of aseam sensor with the operations of a robotic system controlling aneffector end in relation to the seam comprising the followingsteps:sensing successive discrete locations on the seam ahead of theeffector end while concurrently storing data representing seamcoordinates at each of said seam locations as a function of the elapseddistances from the first of such seam locations; modeling the storeddata to generate a model representing the coordinates of said seamlocations as a function of the elapsed distance; deriving from therobotic system a representation of the effector end coordinates forsuccessive discrete locations thereof; calculating iteratively from saidderived effector end coordinates an indication of the elapsed distancefor each of the successive effector end locations and from the first ofsuch effector end locations; extrapolating from a present and a lasteffector end location an anticipated effector end location at an elapseddistance therefrom; deriving from the stored data in said model thecoordinates of the effector end at said anticipated effector end elapseddistance; and controlling the robotic system in relation to said derivedeffector end coordinates.